N ATURAL P ROVER : Grounded Mathematical Proof Generation with Language Models
Anonymous Author(s)
AffiliationAddress email
Abstract
Theorem proving in natural mathematical language – the mixture of symbolic andnatural language used by humans – plays a central role in mathematical advancesand education, and tests aspects of reasoning that are core to intelligence.
Yetit has remained underexplored with modern generative models.
We study largescale language models on two new generation tasks: suggesting the next step in amathematical proof, and full proof generation.
We develop N ATURAL P ROVER , alanguage model that generates proofs by conditioning on background references(e.g. theorems and definitions that are either retrieved or human-provided), andoptionally enforces their presence with constrained decoding.
On theorems fromthe N ATURAL P ROOFS benchmark, N ATURAL P ROVER improves the quality ofnext-step suggestions and generated proofs over fine-tuned GPT-3, according tohuman evaluations from university-level mathematics students.
N ATURAL P ROVERis capable of proving some theorems that require short (2-6 step) proofs, andproviding next-step suggestions that are rated as correct and useful over 40% ofthe time, which is to our knowledge the first demonstration of these capabilitiesusing neural language models. to
Introduction
Constructing a rational argument that justifies a claim is a key aspect of explaining, verifying, andcommunicating ideas in situations ranging from everyday interactions, to legal and political discourse,to science and mathematics [9, 41, 23].
Within the latter context, a mathematical proof – a sequenceof logical arguments expressed in a mixture of symbolic and natural language – assumes this role byproviding justification and insight into why a claim is true [11].
Proofs operate on a relatively explicitand objective set of ground knowledge, isolating a subset of reasoning that is desirable for modelsthat form the foundation of machine learning systems [3].
Moreover, we envision assistive systemsthat provide suggested proofs or next-steps, analogous to language-model-based code suggestions(e.g.
GitHub CoPilot [6]) or formal proof assistants (e.g. GPT- f [19]), which could make learning orusing mathematics more productive and accessible.
To this end, we study the capabilities of large-scale language models (e.g. GPT-3 [5]) on two newtheorem proving tasks in natural mathematical language: next-step suggestion , in which a modelsuggests the next step of a proof, and full-proof generation , in which a model fully proves a claim.
Asproofs are grounded in knowledge from past results (e.g. theorems, definitions), analogous to factsdeployed in a conversation [12], prior rulings used in a legal opinion [15], or articles used to justifyan answer [29], we develop a methodology for obtaining and using background knowledge to provetheorems with a generic language model.
We develop N ATURAL P ROVER , a language model that generates proofs by conditioning on background references (e.g. theorems and definitions that are either retrieved or human-provided), andoptionally enforces their presence with a constrained decoding algorithm that leverages the multi-stepstructure of proofs.
On a collection of theorems from the N ATURAL P ROOFS benchmark [44], N ATURAL P ROVER improves the quality of next-step suggestions and generated proofs over fine-tuned
GPT-3, according to human evaluations from university-level mathematics students.
N ATURALP ROVER is capable of proving some theorems that require short (2-6 step) proofs, and providingnext-step suggestions that are rated as correct and useful more than 40% of the time, which is to ourknowledge the first demonstration of these capabilities using neural language models.
Along with these successes, we study deficiencies in our current models.
We find that modelscan struggle with logical coherence on longer proofs, with providing valid justifications, and withperforming multi-step symbolic derivations.
Taken together, our tasks, methodology, and evaluationshow the feasibility of language models as interactive aids in mathematics, along with open challenges.
N ATURAL P ROOFS -G EN Dataset and Tasks
We create a N ATURAL P ROOFS -G EN dataset adapted from N ATURAL P ROOFS [44], and use thedataset for two tasks: suggesting the next step of a proof, and fully proving a theorem.
N ATURAL P ROOFS -G EN .
N ATURAL P ROOFS -G EN adapts data from N ATURAL P ROOFS , whichcontains theorem statements, proofs, definitions, and additional pages (e.g. axioms, corollaries)sourced from ProofWiki, an online compendium of community-contributed mathematical proofs.
In
N ATURAL P ROOFS -G EN , each example ( x
, y ) ∈ D pairs a theorem x with a gold proof y , both ofwhich are a mixture of text and L A TEX.
[44] split the examples and reference sets into training, dev,and test splits to ensure that no theorem in the dev or test splits was mentioned in the training split.
We adopt these splits of roughly 12.5k training, 1k validation, and 1k test examples, and sampledcore evaluation sets with 100 dev and 100 test theorems that are used for human evaluation.
Theproofs contain additional structure, discussed next.
Multi-step proof structure.
Each proof has a multi-step structure, meaning that a proof y =( y 1 , . . .
y | y | ) is a variable-length token sequence that is segmented into proof steps , where each stepy t is itself a variable-length sequence of tokens
References.
Each proof mentions a variable-number of references { r 1 , . . .
, r R y } from a set R ofroughly 33k theorems and definitions, analogous to how Wikipedia articles reference other pages.
For example, Figure 1 shows a proof with reference mentions in blue.
Each mention identifies a reference by its title and provides a natural language surface form.
For instance, in Figure 1, thefirst proof step mentions the definition of even integer as even, which is formatted in the proof as[[Definition:
Even_Integer|even]] and tokenized along with the rest of the proof.
Tasks.
We consider two tasks that are motivated by an assistive system that provides suggestedproofs or next-steps to a user.
The full proof generation task is to generate a proof y given a theoremx .
The next-step suggestion task is to generate a set of next steps { y kt } Kk =1 given theorem x andproof history
y <t from a gold proof.
In each case, we consider an additional provided referencesetting where the model is also given the set of references { r ∗ 1 , . . .
, r ∗ R y } from a gold proof of thetheorem.
The next-step task simulates a human correctly proving the theorem up to a point, thenquerying a system for suggested next-steps when stuck, while the provided reference setting simulatesa human specifying a plan for a system that writes a proof.
Grounded Proof Generation via Language Modeling
We describe N ATURAL P ROVER , a language model which generates grounded proofs by conditioningon references and optionally enforcing their presence with constrained decoding.
Setup.
Our objective is to generate correct proofs, ˆ y = arg max y correct( x , y ) .
Unfortunately,evaluating proof correctness is costly, and is only done once at test time.
A naive approach is toapproximate the objective, ˆ y ≈ arg max y log p θ ( y | x ) , by fine-tuning a language model p θ on ( x , y )examples and using a decoding algorithm (e.g. greedy decoding).
We instead investigate conditioningon background knowledge in the form of reference documents, p θ ( y | x , R ) , which is beneficialin related generation settings (e.g. [37]), and offers control over the generated proof.
To do so,
N ATURAL P ROVER uses in-context references and a reference reconstruction objective.
In-context references.
Language models have a limited context window that prevents conditioningon full documents.
Instead, N ATURAL P ROVER conditions on a set of reference titles, p θ ( y | x , R title ) .
Concretely, we fine-tune on (theorem, reference titles, proof) sequences of the form,
[Equation]
) with new-lines and {} tokens omitted, relevant strings inserted, and loss only on tokens after <proof> .
Reference reconstruction.
Reference titles do not capture all of the information contained in thereference documents.
We learn a mapping between each reference title and its underlying documentwith a reference reconstruction objective, p θ ( r | r title ) for references r in the training reference set.
Concretely, we fine-tune on additional (title, content) pairs of the form,
[Equation]
where the {type} is theorem/definition/other, and the loss is only on tokens after <content> .
Intuitively,this lets the model associate each reference title with the reference’s underlying content.
The joint objective.
For training, we minimize the joint loss,
[Equation]
Evaluation-time references.
We consider two settings for evaluation-time references: (i) retrievedreferences, from a retrieval model f ( x ) → { r 1 , . . .
, r k } , and (ii) human-provided references fromthe ground-truth proof.
The retrieval setting simulates a fully automated proof assistant, while thesecond simulates a human specifying a plan for an assistant that writes a proof, and acts as an upperbound for a retrieval system optimized to predict references in a ground-truth proof.
3.1 Stepwise constrained decoding
In the provided-reference setting, the conditioned references are known to be relevant to a correctproof.
We hypothesize that that explicitly encouraging generated proofs to contain the references willimprove correctness, by placing lexical constraints on the reference-titles at decoding time,
[Equation]
subject to
[Equation]
where I [ · ] is an indicator function.
To approximate this objective, we generate step-by-step bysampling multiple proof-step candidates, retaining those with high value (reference coverage andlog-probability) in a beam, and continuing to the next step, which we call stepwise beam search.
Value function.
The search supports any function of the proof-so-far, v ( y ≤ t )
→ R .
We use a valuefunction that is a weighted combination of constraint satisfaction and log-probability,
[Equation]
where v constraint ( y ≤ t ) is the number of unique in-context reference-titles in y ≤ t , and v LM ( y ≤ t ) islog p θ ( y ≤ t ) .
We normalize each term by dividing by the maximum absolute value among candidates.
Stepwise beam search.
The procedure generates a proof y =
( y 1 , . . .
, y T ) by iteratively samplingand pruning next-proof-step candidates y t .
Each iteration expands a size- K beam of proofs-so-far,
S t − 1 = { y k<t } Kk =1 , by generating N next-step candidates,
[Equation]
where q is a decoding algorithm (e.g. temperature sampling) and ◦ is concatenation.
The nextiteration’s beam is formed by selecting the top scoring candidates, S t = arg top - K y ≤ t ∈ S ′ t v α ( y ≤ t ) .
When a proof in the beam terminates, it is not expanded further.
The search ends when the beamconsists of K terminated proofs.
The highest value proof is returned as the final output.
Stepwise++.
We add two mechanisms for promoting exploration at each step.
First, we expand eachprefix in the beam (Eqn. 6) by sampling with multiple temperatures, { y nt ∼ q τ ( ·| y <t , x , R title ) | τ ∈{ τ i } mi =1 } , where q τ is sampling with temperature τ .
This relaxes the commitment to a singletemperature for all proof steps, balancing exploration (higher τ ) with exploitation (lower τ ).
Second, rather than selecting the top-K candidates, we select clusters based on different value weights:
S t = ∪ α ∈{
α j } ℓj =1 top K ′ ( S αt ) , where S αt is the set of candidates scored with v α , and K ′ =
K/ℓ .
Thisinterpolates between selecting steps based on likelihood (low α ) and constraint satisfaction (high α ).
Full proof sampling and greedy decoding.
An alternative is to sample full proofs and select the bestone according to the value function.
This can be viewed as expansion (Eqn. 6) done at the full proof,rather than the step level.
Moreover, greedy decoding corresponds to expanding only 1 candidatewith temperature → 0 .
We formalize this in §E as a segment-level search that contains stepwise++,full proof sampling, and greedy decoding as special cases.
Proof Evaluation
A proof’s correctness is contingent on a variety of factors, including reasoning with past results,performing symbolic derivations, and altogether providing sufficient evidence that the claim is true.
We design a human-evaluation schema that isolates these aspects at the proof-step level, along with afull-proof summary.
Table 1 summarizes the schema, which we overview below.
References.
First, proofs involve deploying statements from references, such as applying a definitionor adapting it to fit the context.
Deployments should be consistent with the reference, e.g. deployingthe definition of even integer as ‘...by definition,
∃ k ∈
Z : x = 2 k ...’, rather than ‘... ∃ k ∈ Z : x =2 k + 1 ’, and are a common source of errors in student proofs [14].
Second, proofs use references as justification for steps of reasoning; for instance, Real Addition is
Commutative provides justification for the statement x + y
=
y
+ x where x, y ∈ R , but not forxy = yx .
This aspect is analogous to using an article to justify a claim (e.g. [29]).
Finally, proofsshould not hallucinate references, or ‘beg the question’ by self-referencing.
Equations.
Proofs contain a variety of multi-step derivations, ranging from simple arithmetic tomore sophisticated derivations (e.g. see Table 17).
A derivation should start with a valid equationgiven the surrounding context (e.g. x + x = 2 x in Table 1 versus x + x = 3 x ).
Each subsequent stepshould be a valid derivation from the previous step, e.g. stating =
(2 k + 6) − 1 after y = 2 k + 5 .
Other reasoning , language , & symbolic errors.
A proof should provide sufficient evidencethat a claim is true to a human reader; it should not skip steps.
Proof steps should make progresstowards proving the goal; in particular, they should not repeat known conditions in the theorem orconclusions made in a prior step.
Finally, our schema leaves room for any other reasoning errors, aswell as symbol errors (e.g. undefined symbols) and language errors (e.g. incomplete statements).
[Table]
Since x is an even integer, ∃ k ∈ Z : x = 2 k + 1 .
[Table]
Justification
E ( X 2 ) =
(cid:80) nk =1 k 2 Pr ( X = k )
Power Series for Exponential Function
Hallucinated Ref.
From
[Table]
Number are Irrational , 3 √ 2 is irrational.
Self
[Table]
(Proving Pythagoras’s Theorem:)
From Pythagoras’s Theorem, c 2 = a 2 + b 2 .
Reasoning
[Table]
Equation
[Table]
∀ x ∈ R , x + x = 3 x .
[Table]
(Since x is an even integer, x + 1 = 2 r + 1 ) =
2( r + 1)
Reasoning
[Table] [Table]
is not
[Table]
[Table]
Therefore, x 3 ≡ 1 or
[Table]
Repetition (Let △ ABC be a right triangle.)
Then △ ABC is a right triangle.
[Table]
is an even integer.)
So, x + 1 is an even integer.
Language
Let c = √ a 2 \add b 2 be the ( incomplete statement ; unknown symbol \ add )
Symbolic (Let x ∈ R .)
Let y = x ◦ x − 1 .
( undefined operator ◦ for real numbers )
Usefulness and correctness.
To judge the potential utility of language models as assistive systemsin natural mathematics, we measure whether generated next-steps and full proofs are potentiallyuseful hints for proving the theorem on one’s own.
Additionally, we measure a summary judgment ofcorrectness.
Note that an incorrect statement can still be helpful; for instance, it could give a hint forthe type of reference to use, derivation to perform, argument to make, etc.
Human evaluation protocol.
We measure these aspects through human annotation at a step-wise andan overall level.
For a step-wise annotation, an annotator is presented with the theorem, proof-so-far,and a generated next-step.
The annotator labels the { 0 , 1 } correctness, usefulness, and presence offine-grained errors outlined above.
After labeling each step of a proof, the annotator rates the fullproof’s overall correctness and usefulness on a 0-5 scale.
A rating of 4 or 5 is needed to be consideredas correct, and a rating of 3 or above is needed to be considered as useful.
Automatic metrics: lexical content.
As automatic proxies for quality, we compare each generatedproof against its ground-truth counterpart using the sentence-level n -gram matching metric G LEU[28], and following work in knowledge-grounded dialogue [37] we use F1 overlap between generatedand ground-truth tokens.
Prior to computing the metrics, we normalize the generated and groundtruth proofs by only keeping the surface form of references, removing formatting characters with a
MediaWiki parser, and collapsing any consecutive whitespace into a single space.
Automatic metrics: knowledge grounding.
We define knowledge grounding as meaning that agenerated proof contains the same references as those found in the ground-truth proof.
To measurethis, we use precision, recall, and F1-score between the reference sets contained in the generated andground-truth proofs; i.e. m ( { ˆ r 1 , . . .
, ˆ r ˆ R } , { r ∗ 1 , . . .
, r ∗ R ∗ } ) , where m ( · ) is precision, recall, or F1.
Wealso use Knowledge Token-F1 (kF1) ([37]), the overlap of the generated proof’s tokens with tokenscontained in the references mentioned in the ground-truth proof.
Experiments
We use the training and dev splits of N ATURAL P ROOFS -G EN during fine-tuning, and the coreevaluation sets consisting of 100 theorems from the validation set and 100 from the test set forevaluation (see §2).
These theorems were selected by the authors such that by looking at the theoremtitle each author could recall its content and sketch a proof.
While this may shift the evaluationtowards an easier slice of the dataset, it was necessary to make human evaluation at a meaningfulscale feasible.
We also use the core sets for explorations and ablations.
We finetune three GPT-3 [5] (Curie) models, using the OpenAI API (see Appendix F for details):
[Table]
Baseline GPT-3.
We finetune a baseline GPT-3 model, p θ ( y | x ) , on theorem-proof examples{ ( x , y ) } from the training split.
At test time, we condition the model on a test theorem.2.
N ATURAL P ROVER R ETRIEVE .
We finetune GPT-3 with retrieved references, p θ ( y | x , ˆ r 1 , . . .
, ˆ r 20 ) .
We use a pretrained joint retrieval model f ( x ) → ( r 1 , . . .
, r |R| ) from [44], which was trained toretrieve an input theorem’s ground truth references.
At test time, the model receives a theoremand the top-20 reference titles that are retrieved given the theorem.3.
N ATURAL P ROVER
We finetune GPT-3 with human-provided references, p θ ( y | x , r ∗ 1 , . .
, r ∗ R y ) ,where { r ∗ 1 , . . .
, r ∗ R y } is the set of reference-titles in the ground-truth proof.
We use reference-titleconditioned examples (Eqn. 1) and reference-reconstruction (Eqn. 2) on the training split/referenceset.
At test time, the model receives a theorem and reference titles from its ground-truth proof.
For next-step suggestion we use the human-provided knowledge model (N ATURAL P ROVER ).
Decoding.
For full proof generation, we use stepwise++ decoding with the provided knowledgemodel, which we refer to as N ATURAL P ROVER ++ , and otherwise use greedy decoding.
We do notuse stepwise constrained decoding with retrieved references since these references introduce noisyconstraints, nor for next-step prediction since the algorithm is designed for multi-step proofs.
See §Ffor additional experimental details.
Human evaluation setup.
To evaluate the proofs generated by N ATURAL P ROVER , we recruited15 students from the Department of Mathematics and Applied Mathematics at the University ofanonymized , including undergraduate, masters, and Ph.D. students.
The annotators were trained onhow to evaluate proof correctness and compensated according to IRB requirements; see §G.2.
Foreach task, we first reveal the theorem and its gold proof to the annotator.
If they cannot understanda theorem or its gold proof, they may skip evaluating it.
Otherwise, they may proceed to see themodel-generated proof, one step at a time, and annotate each step under the step-wise evaluationschema (outlined in §4).
After all the steps are shown and evaluated, for the full-proof generationtask, the annotator is asked to annotate the entire proof under the overall evaluation schema.
5.1 Main Results
Our best method is capable of generating correct and useful proofs.
According to humanevaluation results (Table 2), our best method is N ATURAL P ROVER with human-provided referencesand stepwise++ inference.
32% of the proofs generated by this method are rated as correct and 45%are rated as useful as an aid for human proof writers.
On the per-step level, 35% of the proof stepsare correct and 47% are useful .
Taking a more granular view, our best method makes significantlyless reference , equation , and other reasoning errors than other baselines.
It makes very fewlanguage and symbolic errors , meaning it produces mostly complete, well-formatted mathematicalstatements, and defines and uses symbols accordingly.
It mostly avoids hallucinating references orcreating circular proofs.
On the other hand, despite improving over the other methods, the modeloften struggles with correctly deploying and using references (23.6% reference error rate), as well assymbolic computations (28.5% equation error rate), especially multi-step derivations (21.9% invalid).
What do the model’s correct proofs look like?
We inspected the proofs labeled as correct andfound three main categories: (1) reference-assembly proofs whose correctness is heavily determinedby reference statements (e.g. Table 18, Table 20); (2) template-adaptation proofs in which the modeladapts the structure and content of a training theorem’s proof to prove the unseen evaluation theorem
[Table]
(e.g. Table 21, Table 22); (3) complex proofs that are not fully determined by reference statements anddiffer significantly from training proofs (e.g. Figure 1, Table 3).
In terms of techniques, our methoddemonstrates some ability to produce direct proofs (Table 19), proofs by cases (Table 22), proofs byinduction (Table 23), utilize references (Table 20) and do symbolic computations (Table 21).
Vanilla fine-tuned GPT-3 struggles with proof generation.
The vanilla fine-tuned GPT-3 modelyielded fewer useful and correct proofs, with more reference-based and other reasoning errorsthan all three knowledge-grounded settings.
The model showed severe reference hallucination (18%)and repetition (23%).
It also makes significantly more reasoning errors related to reference usage.
Language and symbolic error rates roughly stay the same.
Overall, naively fine-tuning GPT-3 ontheorem-proof examples alone is suboptimal for proof generation.
Human-provided knowledge improves proof generation.
Grounding the generations with humanprovided references significantly raises correctness and usefulness of the proofs in both fullproof and per-step evaluation.
It most substantially reduces reference errors , especially invaliddeployments and hallucinated references.
For example, Table 3 shows the model grounding a proofwith information from the theorem Singleton Point is Isolated and the definition of Dense-in-itself, incontrast to the vanilla GPT-3 model which hallucinates references.
Retrieved knowledge also improves proof generation.
Retrieved knowledge also turns out to bevery helpful, and even comparable to human-provided knowledge in some metrics.
Although theretrieval model is far from perfect, the proof generation model is capable of narrowing down theretrieved reference titles provided in its context, assembling proofs that are useful and correct moreoften than the no-knowledge model.
Qualitatively, we found examples where grounding in retrievedreferences eliminates repetition, enables multi-step derivations justified by references (Table 21),and assembles references into a correct proof (Table 20).
This paves a promising path towards fullyautomated mathematical proof generation in natural mathematical language.
[Table]
the constrained objective better than greedy decoding, resulting in both lower perplexity and better reference coverage, regardless of whether knowledge is provided in-context.
Constrained decoding further improves proof generation.
Table 4 confirms that stepwise++ decodingapproximates the constrained objective (Eqn. 4) better than greedy search, yielding proofs with lowerperplexity and higher constraint satisfaction (Ref-F1).
This translates to generations that are correct anduseful more often according to the annotators.
Intuitively, the constraints encourage the model to includereferences that help prove the claim (e.g. Table 18).
Next-step suggestion.
The next-step suggestiontask characterizes a model’s performance on making a single proof step given a correct proof-so-far.
In Table 2 we use the provided-knowledge model with greedy decoding for next-step suggestion, and find that reasoning errors decrease and per-step usefulness and correctness improvecompared to the full proof setting, with 51% of the proof steps rated as useful and 43% correct.
Although we used a single suggestion in our human evaluation study, in Table 5 wesimulate a user choosing from among multiple suggestions by sampling 10 next-steps fromour model and computing automatic metrics on the sample with the best sum of metrics.
Using 10 samples instead of greedily decoding a single sequence substantially improves eachmetric, suggesting that utility might be increased further by presenting multiple suggestions.
[Table]
[Table] [Table]
[Table]
Sampling 10 suggestions improves over a single greedy suggestion.
How good are Automatic Metrics?
We study how well theautomatic lexical and grounding metrics introduced in (§4) canreflect the real quality of proofs, as a guide for using them asa proxy evaluation protocol for N ATURAL P ROOFS -G EN .
Wecompute the Pearson correlation coefficient between each pairof human and automatic metrics, with data from the four experiment settings for full-proof generation.
Results are shown inthe lower part of Table 6, with error metrics negated, meaningpositive correlation is desired.
The lexical and grounding metrics positively correlate with fullproof correctness and usefulness ( ≥ 0.8).
At the step-level, the metrics show (i) high correlationwith step-level correctness and language errors ; (ii) varied, but positive, correlations with aggregatereasoning errors; (iii) negative correlation with symbolic errors (though symbolic errors are relativelylow for all models).
The results suggest that optimizing for automatic metrics may be a viable strategy,albeit without guarantees on how finer-grained reasoning aspects vary across proofs.
5.2 Ablations and error analysis.
Recon.
Gleu Ref
[Table]
F1 Halluc
[Table]
Reference reconstruction.
We fine-tune an additional GPT-3model that is provided with in-context reference titles, butwithout reference reconstruction.
As seen in Table 7, reference reconstruction improves content and reference usage.
Constrained decoding.
First, Table 9 compares the step-levelsearch in stepwise++ with searching at the full-proof levelthrough sampling multiple proofs and selecting the best withthe N ATURAL P ROVER value function ( rerank (n) ).
Reranking 60 samples matches the cost of stepwise++ in terms ofnumber of decoded tokens.
Full-proof reranking yields the best Gleu, though with lower reference-F1.
Second, Table 8 shows that the expansion and selection mechanisms together result in the bestreference matching, while holding Gleu at a similar level.
Finally, Table 13 shows that both terms in the N ATURAL P ROVER value function αv constraints + (1 − α ) v LM are needed: increasing the constraintweight α increases reference-matching, with a tradeoff in Gleu at high values.
Language model comparison.
Table 10 varies the language model used to parameterize N ATURAL P ROVER .
The content and reference usage metrics improve with larger models.
Separately, wefind that increasing inference-time compute closes the gap in reference-matching between GPT-2and
the larger GPT-3 model (Table 11): sampling 10 full-proofs from GPT-2 and selecting the bestusing the N ATURAL P ROVER value function achieves the same reference-F1 as GPT-3 with a singlegreedily-decoded proof.
However, Gleu remains much higher with the larger GPT-3 model.
Challenge: Reasoning with references.
Although reference reasoning errors were decreased throughknowledge-grounding and constrained decoding, N ATURAL P ROVER still commits a reference erroron 23.6% of test steps (27% dev), with 15% of steps containing invalid deployments and 10% invalidjustifications.
For next-step prediction, the reference error rate remains nontrivial (19.7% test, 13%dev).
, meaning that the model can struggle to correctly deploy references or use them as justificationeven in the absence of compounding errors from previous steps.
Table 15 shows example invaliddeployments and justifications; the errors are at times subtle, and require reasoning about the theoremstatement, reference content, and proof context.
Challenge: Equations and derivations.
N ATURAL P ROVER commits an equation-related error on 28.5% of test steps (22.8% dev), including invalid equations (9.4%) and derivations(21.9%).
Though an improvement over vanilla fine-tuned GPT-3 (32.5%), the errors occur frequently and remain high for next-step prediction (26%).
Table 17 shows representative errors,which range from simple ‘commonsense’ mistakes (e.g. 24 = 2 3 ) to making invalid stepswith false justification within more sophisticated multi-step proofs.
Investigating the role ofpretraining, in-context techniques [30], and autoformalization [38] is interesting future work.
Challenge: Proof length.
Although N ATURALP ROVER demonstrates some ability to write longproofs (e.g. Table 23), the 42% next-step correctnesssuggests that compounding errors are likely as prooflength increases.
Indeed, our best model’s full-proofcorrectness is 48% on 1-4 step proofs ( n = 102 ),decreasing to 15.6% on proofs with 5 or more steps( n = 64 ), with lower per-step usefulness and correctness at later steps (Figure 2).
Our findings areanalogous to recent work on language modeling forformal theorem proving [31], where current modelsare typically limited to chaining 2 or 3 non-trivialsteps of mathematical reasoning.
Related Work
Language models have been used for interactive theorem proving in formalized mathematics, inwhich proofs steps are verifiable and written in a programming language [32, 31, 19, 40].
Previouswork on informal theorem proving focuses on premise retrieval [16, 17, 44, 20], or informal-to-formaltranslation [42], which differs from generating next-steps or full proofs.
Outside of theorem proving,language models are used for problem solving [36, 21, 7, 43], math-related program synthesis[24, 2, 13] and competitive programming [25].
These settings generate executable programs ora numerical answer, which differ from our theorem proving setting, where the goal is generatingarguments on many topics in natural mathematical language.
See Appendix C for more related work.
Conclusion
We described N ATURAL P ROVER , a knowledge-grounded language model that generates mathematicalproofs by conditioning on background theorems and definitions, and optionally enforces their presencewith constrained decoding.
Our system improves the quality of next-step suggestions and generatedproofs over fine-tuned GPT-3, demonstrating an ability to correctly prove theorems and provideuseful suggestions to human proof writers.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See §5.2.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See §H.(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We will release our code inthe supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
[Yes] See §D and §F.(c)
Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?
[Yes] See §F.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[Yes] See §D.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See§G.2 and supplementary materials.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See §G.2.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See §G.2.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica-
ble?
[Yes] See §G.2 and supplementary materials.
(b) Did you describe any potential participant risks, with links to Institutional Review Board
(IRB) approvals, if applicable?
[Yes] See §G.2.
(c) Did you include the estimated hourly wage paid to participants and the total amount spenton participant compensation?
[Yes] See §G.2.
A Additional ResultsA.1 Additional ablations
Table 10 shows automatic metrics with various language models used to parameterize N ATURALP ROVER .
Table 11 shows results with the 774M parameter GPT-2 model with greedy decoding, and full-proofsampling & reranking with 5 and 10 samples, compared to the 13B parameter GPT-3 with greedydecoding.
We use τ = 0 .
3 and α = 0 .
75 based on our full-proof sampling experiments with GPT-3.
Table 13 varies the value function parameter α (core dev set).
We use full-proof sampling sincestepwise++ uses multiple values of α in its selection.
[Table] [Table]
value function reference-matching gap
[Table]
[Table]
search corresponds Constrained stepwise++ decoding substantially improves grounding metrics compared to stochastic beam search, while keeping the lexical content metrics at a similar level. Core validation set.
A.2 Multiple next-step suggestions
Table 14 shows next-step suggestion results with 10 sampled suggestions versus greedy decoding. Lexical Grounding
Decoding
[Table]
B Qualitative Examples
B.1 Error Analysis
B.1.1 Reference errors
Theorem
Geometric Mean of two Positive Real Numbers is Between them Let a,b ∈ R be real numbers such that 0 < a < b .
Let G ( a,b ) denote the geometric mean of a and b .
Then: a < G ( a,b ) <
b
N ATURAL P ROVER ++ Proof: correct deploy By definition of geometric mean:
G ( a,b ) =
a 2 + b 2✗ ✗ ...
Gold Proof
Proof: By definition of geometric mean:
G ( a,b ) := √ ab where √ ab specifically denotes the positive square root of a and b . ...
Reference
Geometric Mean Let x 1 ,x 2 ,...,x n ∈ R > 0 be (strictly) positive real numbers.
The geometric mean of x 1 ,x 2 ,...,x n is defined as:
[Equation] n .
Theorem
Power Set of Transitive Set is Transitive Let S be a transitive set.
Then its power set P ( S ) is also a transitive set.
N ATURAL P ROVER ++ Proof: correct deploy From Set is Transitive iff Subset of Power Set:
S ⊆ P ( S ) ✓
From Power Set of Subset: P ( S ) ⊆
S ✗ ✗ ...
Gold Proof
Proof: Let S be transitive.
By Set is Transitive iff Subset of Power Set:
S ⊆ P ( S )
Then by Power Set of Subset: P ( S ) ⊆
P ( P ( S )) ...
Reference
[Table]
[Table]
[Table]
B.1.2 Equation errors
Theorem
Sum of Squares of Divisors of 24 and 26 are Equal
The sum of the squares of the divisors of 24 equals the sum of the squares of the divisors of 26 : σ 2 (24)
=
σ 2 (26) where σ α denotes the divisor function.
Proof: correct eqn.
N ATURAL P ROVER ++ 24 = 2✗ ✗ 26 = 2✗ ✗ ⇝ σ 2 (24) =
σ 2 (26) ✗ ✗ ■
Theorem
Trace of Sum of Matrices is Sum of Traces Let A =
[ a ] n and B =
[ b ] n be square matrices of order n .
Let A + B denote the matrix entrywise sum of A and B .
Then: tr ( A + B ) = tr ( A ) + tr ( B ) where tr ( A ) denotes the trace of A .
Proof: correct eqn.
ref.
N ATURAL P ROVER ++ tr ( A + B ) =
(cid:80) C =
A + B tr ( C )
Trace of Matrix ✗ ✗ ✗ =
(cid:80) C
=
A tr ( C ) + (cid:80) C
= B tr ( C ) Sum of Summations equals Summation of Sum ✗ ✗ ✗ =
tr ( A ) + tr ( B )
Trace of Matrix ✗ ✗ ✗ ■
Theorem
Moment Generating Function of Discrete Uniform Distribution Let X beadiscrete random variablewithadiscrete uniform distribution with parameter n for some n ∈ N .
Then the moment generating function M X of X is given by:
[Equation]
Proof: correct eqn.
ref.
N ATURAL P ROVER ++ From the definition of the discrete uniform distribution ,
X has probability mass function: Pr( X = k )
=
n n + k ✗ ✗ ✗
From the definition of a moment generating function:
M
[Equation]
✓
So: M
[Equation]
✓ = nn + k (cid:80) nk =0 (( e tk ) n (cid:81) n − 1 j =
( e tj +1 n ))
Sum of Geometric Sequence ✗ ✗ ✗ = nn + k (cid:80) nj =
( e tj +1 n e tj ) ✗ ✗ = nn + k (1 − e nt )
Discrete Uniform Distribution ✗ ✗ ✗ = et (1 − ent ) n (1 − et ) ✗ ✗ ■
B.2 Examples
Theorem
Metric Space is Perfectly Normal
Gold Proof Let M =
( A,d ) be a metric space.
By definition, a topological space is perfectly normal space
iff it is: Then M is a perfectly normal space.
perfectly T 4 space and T 1 (Fréchet) space.
We have that:
Metric Space is Perfectly T 4 Metric Space is T 2 (Hausdorff)
T 2 (Hausdorff) Space is a T 1 (Frechet) Space.
■
N ATURAL P ROVER ++ From:Metric Space is Hausdorff
N ATURAL P ROVER (without stepwise++)
From:Metric Space is Perfectly T
T 2 (Hausdorff) Space is T 1 Space Metric Space is Perfectly Tit follows that M is a topological space which is perfectly normal.
Metric Space is T 2 (Hausdorff) it follows that M is a perfectly normal space .
[Table]
[Equation] [Table]
[Table] [Table]
Title
[Table]
Theorem Content:
Gold Proof
[Table]
C Expanded Related Work
Formalized mathematics with neural language models.
A large portion of work on machinelearning for mathematics focuses on formalized mathematics.
Language models have been used forinteractive theorem proving, including in GPT- f [32, 31], PACT [19], and in [40].
In these settingsproof steps are expressed in a programming language (e.g. Lean [10]) and there is access to a verifier,which differs from our setting of theorem proving in natural mathematical language.
Informal mathematics with neural language models.
Previous work on theorem proving innatural mathematical language focuses on retrieving relevant premises (e.g. theorems, definitions)[16, 17, 44, 20], or informal-to-formal translation [42], which differ from our setting of generatingnext-steps or full proofs.
Outside of theorem proving, various works use sequence models for problemsolving, including benchmarking language models on arithmetic [36] or competition problems [21],symbolic mathematics [24, 45], augmenting LMs with verifiers [7] or in-context rationales [43]for math word problems, or using language models for math-related program synthesis [2, 13]and competitive programming [25].
These settings focus on generating executable programs or anumerical answer, which differ from our theorem proving setting, where the goal is to generate soundand convincing arguments on a range of topics in natural mathematical language.
Related areas in NLP.
Systematic reasoning in natural language (outside of math) has been studiedwith synthetic proofs [35, 39], single-step deductions [4], or entailment trees [8], which differ fromproving real-world mathematical theorems.
Augmenting LMs with knowledge reduces hallucinationsin dialogue [37] which has an analogous step-wise structure, while [29] use references within longform answers; these and related NLP findings differ from improving the utility of mathematicalproofs.
Lexically-constrained decoding algorithms include variants of (token-level) beam search(e.g.
[1, 22, 27, 26]) which assume access to per-token logits, and gradient-based decoding [33]; oursegment-level decoding only assumes a sampler that returns text and its log-probability
(e.g. the
GPT-3 API).
D Dataset Details
We provide an overview of N ATURAL P ROOFS and its ProofWiki domain from which we build
N ATURAL P ROOFS -G EN .
Refer to [44] for further details about N ATURAL P ROOFS .
Our dataset is derived from N ATURAL P ROOFS , a multi-domain corpus of theorem statements, proofs,definitions, and additional pages (e.g. axioms, corollaries) in natural mathematical language.
Weuse the ProofWiki 2 domain, which provides broad-coverage of many subject areas (e.g. Set Theory,
Analysis) sourced from ProofWiki, an online compendium of community-contributed mathematicalproofs.
P ROOF W IKI contains ∼ 20k theorems, ∼ 20k proofs, ∼ 12k definitions, and ∼ 1k additionalpages (e.g. axioms, corollaries).
The set of all ∼ 33k theorems, definitions, and additional pages formthe reference set R .
Finally, ∼ 14.5k of the theorems x are paired with at least one proof y to formexamples D = { ( x , y ) i } Ni =1 .
[44] split the reference sets and examples into training, validation, andtest splits to ensure that no theorem in the validation or test splits was mentioned in the training split.
E Segment-level Constrained Decoding
In this section we present a generic segment-level decoding algorithm that contains stepwise++,full-proof sampling, and greedy decoding as special cases.
We generate a multi-step proof using avalue function v ( · ) that measures language quality and constraint satisfaction.
Search can be doneat the step-level, in which candidate next-steps are generated and high-value steps are retained ina beam, or at the proof-level, in which multiple proofs are generated and the highest-value proofis selected.
We formalize these into a generic segment-level search , where a segment s t is either aproof-step y t or a full proof y .
The search iteratively builds a multi-step proof y =
( y 1 , . . .
, y T ) by expanding , scoring , and selectinga set of candidate segments :
• Expand : S t − 1 → S ′ t extends segments S t − 1 = { s ≤ t } into candidates S ′ t
= { ( s ≤ t , s t ) } .
The is under the CC BY-SA 4.0 license.
• Score : ( s ≤ t , v ) → R scores a candidate using a value function, v ( s ≤ t ) → R .•
Select : S ′ t → S t prunes candidates S ′ t into segments S t used in
the
next iteration.
Value function.
We score candidates based on constraint satisfaction and language quality,
[Equation]
where v constraint ( y ≤ t ) is the number of unique in-context reference-titles in s ≤ t , and v LM ( s ≤ t ) islog p θ ( s ≤ t ) .
We normalize each term by dividing by the maximum absolute value among candidates.
Greedy search.
This baseline search defines a segment as a full proof, meaning s 0 is an emptysequence and s 1 is a proof y .
Expand samples one segment candidate with temperature 0.
Scoreand select are trivial since there is only one candidate.
Greedy search costs T steps of tokens.
Sample-and-rerank.
In this search, a segment is again full proof, but expand samples N candidates, S ′
= { y n ∼ q ( ·| x ) } Nn =1 , where q is a decoding algorithm (e.g. temperature sampling).
Select takes the top scoring candidate, y = arg max y n ∈ S ′ 1 v ( y n ) .
The cost is NT steps of tokens.
Step-wise stochastic beam search.
This search generates by iteratively sampling and re-rankingnext-step candidates.
In this case, a segment is a proof step, y t , and each iteration starts with abeam of proofs-so-far, S t − 1 = { y k <t } Kk =1 , where K is the beam size.
Expand samples N next-stepcandidates for each proof-so-far in the beam,
S ′ t = (cid:91)
[Equation]
where q is a decoding algorithm (e.g. temperature sampling) and ◦ is concatenation.
Select formsthe next beam using the top- K scoring candidates,
[Equation]
When a proof in the beam terminates, it is not expanded further.
The search ends when the beamconsists of K terminated proofs.
The highest scoring proof is returned as the final output.
The cost is
NTK steps of tokens.
Stepwise++.
At certain proof steps it is important to enumerate and explore options, while at others(e.g. derivations) a single highly probable prediction is better.
To this end, we expand by samplingwith multiple temperatures, meaning that we expand each prefix y <t in (6) using:
[Equation]
where q τ is sampling with temperature τ .
This relaxes the commitment to a single temperature for allproof steps, intuitively balancing exploration (higher τ ) with exploitation (lower τ ).
Second, during the search we want to balance selecting proof steps that satisfy constraints and proofsteps with high log-probability.
To this end, we select clusters with different value weights,
[Equation]
where S α means the set of candidates scored with v = αv constraint + (1 − α ) v LM , and K ′ =
K/ℓ .
This interpolates between selecting steps with good language score ( α small), constraint score ( αlarge), and balance ( α : 0.5).
F Implementation Details and Experimental Setup
Data preprocessing.
We automatically infer the boundaries of proof steps within the raw proofcontents, and merge contiguous lines into atomic proof steps when appropriate.
Steps are separatedby the \n token ( \\n in Python string), and lines within a step are separated by the newline token ( \nin Python string).
Implementation details.
All N ATURAL P ROVER and GPT-3 models are fine-tuned with the OpenAI
API 3 for 4 epochs with a batch size of 64.
Other models (GPT-2/J/Neo) are trained on one Quadro
RTX 8000 GPU.
During inference, the prompt (up to <proof> ) is truncated to 1024 tokens.
Forfull proof generation, we allow a maximum of 1020 generated tokens.
For next-step suggestion, wetruncate the proof-so-far to 900 tokens, and allow a maximum of 120 generated tokens per step. Additional model details.
All GPT-3 models are fine-tuned instances of the Curie engine, thesecond largest model available through the OpenAI API at the time of writing.
The model’sperformance on the EleutherAI evaluation harness 5 is between the 6.7B and 13B variants of theautoregressive transformer language model GPT-3 from [5], 6 though further details of the Curiemodel are not publicly available.
Separately, we fine-tune GPT-J 6B, 7 a publicly available autoregressive transformer language modeltrained on the Pile [18], GPT-2 [34], an autoregressive transformer language model trained on scrapedweb documents, and GPT-Neo-125M, 8 a GPT-2 like causal language model trained on the Pile.
Our retrieval model is the joint retrieval model from [44] trained for reference retrieval on ProofWikiusing the same dataset splits as NaturalProver. We use the publicly-available pretrained model fromthe GitHub repository of [44] and do not update the model further. We use the model to retrieve thetop-20 references for each input theorem. Stepwise++ decoding.
For expansion with multiple temperatures, we use N = 10 candidatessampled with ( n, τ ) ∈ { (1 , 0 . 0) , (3 , 0 . 3) , (3 , 0 . 5) , (3 , 0 . 7) } .
We also tried including τ = 1 . 0 whichresulted in very poor G LEU , and {(1,0.0), (5,0.3), (4,0.5)}.
For selection, we use a beam size K = 9 ,and three equally-sized clusters formed with α ∈ { 0 . 1 , 0 . 5 , 1 . 0 } .
We also tried { 0 . 5 , 0 .
Weuse α = 0 .
75 to pick select the final sequence, based on our ablation with full-proof sampling.
Full proof sampling.
We use temperature τ = 0 . 3 , selected based on a search over τ ∈{ 0 . 1 , 0 . 3 , 0 . 5 , 0 .
7 } using GLEU plus Ref-F1 on the core dev set.
G Additional Evaluation DetailsG.1 Full Evaluation Schema
Table 24 shows the full schema of human evaluation.
The overall correctness and usefulness arerated on a 0-5 scale.
The step-wise correctness and usefulness are yes/no questions, while the errortypes ask for a binary indicator for the existence of each error type.
G.2 Additional Human Evaluation Details
Process.
The authors conducted and moderated group sessions with the annotators.
Each sessionconsisted of 30-minutes of training and a 1-hour working/Q&A period.
After attending the session,annotators could continue working on their assigned tasks for two weeks.
Each annotator wasassigned 25 theorems (with 5 proofs per theorem, equaling 125 total tasks) and asked to completeas many tasks as they would like.
The evaluation guideline that the annotators referenced to can befound in the supplementary materials.
The pre-recorded training video is available at anonymized .
Interface.
We developed an interface that displays theorems and proofs in a rendered, humanreadable format and collects annotations.
The interface is built on MediaWiki 9 , which also powersthe ProofWiki website 10 .
We also developed a web console that helps human annotators navigateannotation tasks and track progress.
Figure 3 shows screenshots of the interface.
Payment.
Human annotators are paid based on the number of tasks they complete.
Each task isworth ($1 .
+ #steps × $0 . 4) .
We pay each annotator an additional $40 for attending the group session.
Annotators are guaranteed a minimal rate of $20/hour.
The human evaluation costs approximately$5,000.
Ethics review.
The human evaluation study is approved by anonymized under IRB
STUDY00014751.
Consent was obtained from each human annotator by signing a consent form via
[-50% of the total evaluations is covered by roughly the top 3 or 
-]
[Table]
◦ 0:
[Table]
proof
[Table]
missing.
◦ 1: The proof makes no sense or is unrelated to the problem statement.
◦ 2: The proof contains serious logical flaws and lacks adequate justification or explanation.
◦ 3: The proof has some gaps in reasoning.
◦ 4: The proof is correct or nearly correct and logically coherent.
[Table]
5: The proof is correct and flows logically.
[Table]
Even if the proof is not perfect, would it be useful to you if you were to prove this theorem?
[Table]
[Table]
proof
[Table]
as
[Table]
solution.
[Table]
Is this step correct? ◦
Yes ◦
No (
[Table]
this if you identified any error in previous questions)
[Table]
Cannot
[Table] [Table]
[Table]
Could this step be a helpful hint for proving the theorem by myself? ◦
Yes ◦
No
Reasoning
[Table]
Reference
[Table]
A statement deployed from a reference is not consistent with the reference.
[Table]
A reference is used as invalid justification for a statement.
[Table]
Ref
[Table]
A reference that does not exist is used.
[Table]
The step refers to the theorem itself.
[Table]
Equation
[Table]
A standalone equation or initial equation in a derivation is invalid.
[Table]
An equation in a derivation does not follow from the preceding steps.
[Table]
Other
[Table]
The step assumes unproven statements, or skips non-trivial steps.
[Table]
The step is merely a repetition of known things.
Invalid (Other)
[Table] [Table]
[Table] [Table]
[Table] [Table]
DocuSign prior to the beginning of study. The IRB approval letter and a template of the consent formcan be found in the supplementary materials. Minimal personally identifiable information (PII) wascollected, and removed prior to any data analysis. G.3 Full results
Table 25 shows the full results of human evaluation, including the error rates of fine-grained errortypes. G.4 Analyzing the Annotators Inter-annotator agreement. We compute inter-annotator agreement using proofs in the core dev setthat get an evaluation from two or more annotators.
Overall, the annotators achieved fair agreement(Fleiss kappa κ = 0 . 24 ). The level of agreement for each evaluation question is shown in Figure 4.
Fair to moderate agreement is reached for identifying coarse-grained error types, while the high-levelquestions (i.e. correctness , usefulness ) have relatively low agreement. Source diversity. Figure 5 shows the largest proportion of evaluations covered by a fixed numberof annotators. The top-1 annotator contributes 20% of the total evaluations when counting by proofsand 18% when counting by steps.
50% of the total evaluations is covered by roughly the top 3 or 4 annotators.
Therefore, our human evaluation results have good source diversity and do not heavilydepend on a single annotator’s opinion.
H Ethical Considerations
Our system may produce proofs of mathematical theorems that are fallacious or misleading, whichmay have negative impact if deployed in real educational environments.
We kindly remind potentialusers that our system and models are experimental, and their outputs should be interpreted critically.