N ATURAL P ROVER : Grounded Mathematical Proof Generation with Language Models
Abstract
Theorem proving in natural mathematical language – the mixture of symbolic and natural language used by humans – plays a central role in mathematical advances and education, and tests aspects of reasoning that are core to intelligence.
Yet it has remained underexplored with modern generative models.
We study largescale language models on two new generation tasks: suggesting the next step in a mathematical proof, and full proof generation.
We develop N ATURAL P ROVER , a language model that generates proofs by conditioning on background references (e.g. theorems and definitions that are either retrieved or human-provided), and optionally enforces their presence with constrained decoding.
On theorems from the N ATURAL P ROOFS benchmark, N ATURAL P ROVER improves the quality of next-step suggestions and generated proofs over fine-tuned GPT-3, according to human evaluations from university-level mathematics students.
N ATURAL P ROVER is capable of proving some theorems that require short (2-6 step) proofs, and providing next-step suggestions that are rated as correct and useful over 40% of the time, which is to our knowledge the first demonstration of these capabilities using neural language models.
36th
Introduction
Constructing a rational argument that justifies a claim is a key aspect of explaining, verifying, and communicating ideas in situations ranging from everyday interactions, to legal and political discourse, to science and mathematics [9, 41, 23].
Within the latter context, a mathematical proof – a sequence of logical arguments expressed in a mixture of symbolic and natural language – assumes this role by providing justification and insight into why a claim is true [11].
Proofs operate on a relatively explicit and objective set of ground knowledge, isolating a subset of reasoning that is desirable for models that form the foundation of machine learning systems [3].
Moreover, we envision assistive systems that provide suggested proofs or next-steps, analogous to language-model-based code suggestions (e.g. GitHub CoPilot [6]) or formal proof assistants (e.g. GPT- f [19]), which could make learning or using mathematics more productive and accessible.
To this end, we study the capabilities of large-scale language models (e.g. GPT-3 [5]) on two new theorem proving tasks in natural mathematical language: next-step suggestion , in which a model suggests the next step of a proof, and full-proof generation , in which a model fully proves a claim.
As proofs are grounded in knowledge from past results (e.g. theorems, definitions), analogous to facts deployed in a conversation [12], prior rulings used in a legal opinion [15], or articles used to justify an answer [29], we develop a methodology for obtaining and using background knowledge to prove theorems with a generic language model.
We develop N ATURAL P ROVER , a language model that generates proofs by conditioning on background references (e.g. theorems and definitions that are either retrieved or human-provided), and optionally enforces their presence with a constrained decoding algorithm that leverages the multi-step structure of proofs.
On a collection of theorems from the N ATURAL P ROOFS benchmark [44], N ATURAL P ROVER improves the quality of next-step suggestions and generated proofs over fine-tuned
GPT-3, according to human evaluations from university-level mathematics students.
N ATURALP ROVER is capable of proving some theorems that require short (2-6 step) proofs, and providing next-step suggestions that are rated as correct and useful more than 40% of the time, which is to our knowledge the first demonstration of these capabilities using neural language models.
Along with these successes, we study deficiencies in our current models.
We find that models can struggle with logical coherence on longer proofs, with providing valid justifications, and with performing multi-step symbolic derivations.
Taken together, our tasks, methodology, and evaluation show the feasibility of language models as interactive aids in mathematics, along with open challenges.
N ATURAL P ROOFS -G EN Dataset and Tasks
We create a N ATURAL P ROOFS -G EN dataset adapted from N ATURAL P ROOFS [44], and use the dataset for two tasks: suggesting the next step of a proof, and fully proving a theorem.
N ATURAL P ROOFS -G EN .
N ATURAL P ROOFS -G EN adapts data from N ATURAL P ROOFS , which contains theorem statements, proofs, definitions, and additional pages (e.g. axioms, corollaries) sourced from ProofWiki, an online compendium of community-contributed mathematical proofs.
In
N ATURAL P ROOFS -G EN , each example ( x
, y ) ∈ D pairs a theorem x with a gold proof y , both of which are a mixture of text and L A TEX.
[44] split the examples and reference sets into training, dev, and test splits to ensure that no theorem in the dev or test splits was mentioned in the training split.
We adopt these splits of roughly 12.5k training, 1k validation, and 1k test examples, and sampled core evaluation sets with 100 dev and 100 test theorems that are used for human evaluation.
The proofs contain additional structure, discussed next.
Multi-step proof structure.
Each proof has a multi-step structure, meaning that a proof y =
( y 1 , . . .
y | y | ) is a variable-length token sequence that is segmented into proof steps , where each step y t is itself a variable-length sequence of tokens (either text or Latex).
The segmentation is largely determined by ProofWiki’s formatting and community standards for structuring proofs, and we additionally merge steps to ensure that each step contains non-trivial semantic content.
For example,
Figure 1 shows a 4-step (generated) proof with each step highlighted in green.
References.
Each proof mentions a variable-number of references { r 1 , . . .
, r R y } from a set R of roughly 33k theorems and definitions, analogous to how Wikipedia articles reference other pages.
For example, Figure 1 shows a proof with reference mentions in blue.
Each mention identifies a reference by its title and provides a natural language surface form.
For instance, in Figure 1, the first proof step mentions the definition of even integer as even, which is formatted in the proof as [[Definition:Even_Integer|even]] and tokenized along with the rest of the proof.
Tasks.
We consider two tasks that are motivated by an assistive system that provides suggested proofs or next-steps to a user.
The full proof generation task is to generate a proof y given a theorem x .
The next-step suggestion task is to generate a set of next steps { y kt } Kk =1 given theorem x and proof history
y <t from a gold proof.
In each case, we consider an additional provided reference setting where the model is also given the set of references { r ∗ 1 , . . .
, r ∗ R y } from a gold proof of the theorem.
The next-step task simulates a human correctly proving the theorem up to a point, then querying a system for suggested next-steps when stuck, while the provided reference setting simulates a human specifying a plan for a system that writes a proof.
Grounded Proof Generation via Language Modeling
We describe N ATURAL P ROVER , a language model which generates grounded proofs by conditioning on references and optionally enforcing their presence with constrained decoding.
Setup.
Our objective is to generate correct proofs, ˆ y = arg max y correct( x , y ) .
Unfortunately, evaluating proof correctness is costly, and is only done once at test time.
A naive approach is to approximate the objective, ˆ y ≈ arg max y log p θ ( y | x ) , by fine-tuning a language model p θ on ( x , y ) examples and using a decoding algorithm (e.g. greedy decoding).
We instead investigate conditioning on background knowledge in the form of reference documents, p θ ( y | x , R ) , which is beneficial in related generation settings (e.g. [37]), and offers control over the generated proof.
To do so,
N ATURAL P ROVER uses in-context references and a reference reconstruction objective.
In-context references.
Language models have a limited context window that prevents conditioning on full documents.
Instead, N ATURAL P ROVER conditions on a set of reference titles, p θ ( y | x , R title ) .
Concretely, we fine-tune on (theorem, reference titles, proof) sequences of the form, <
[Equation]
[Equation] [Equation]
> </theorem
[Equation]
ref
[Equation]
1} </ref
[Equation]
... <ref> {ref
[Equation]
title
[Equation]
R} </ref> <proof> {proof} </proof>
[Equation]
with new-lines and {} tokens omitted, relevant strings inserted, and loss only on tokens after <proof> .
Reference reconstruction.
Reference titles do not capture all of the information contained in the reference documents.
We learn a mapping between each reference title and its underlying document with a reference reconstruction objective, p θ ( r | r title ) for references r in the training reference set.
Concretely, we fine-tune on additional (title, content) pairs of the form,
[Equation]
where the {type} is theorem/definition/other, and the loss is only on tokens after <content> .
Intuitively, this lets the model associate each reference title with the reference’s underlying content.
The joint objective.
For training, we minimize the joint loss,
[Equation]
Evaluation-time references.
We consider two settings for evaluation-time references: (i) retrieved references, from a retrieval model f ( x ) → { r 1 , . . .
, r k } , and (ii) human-provided references from the ground-truth proof.
The retrieval setting simulates a fully automated proof assistant, while the second simulates a human specifying a plan for an assistant that writes a proof, and acts as an upper bound for a retrieval system optimized to predict references in a ground-truth proof.
3.1 Stepwise constrained decoding
In the provided-reference setting, the conditioned references are known to be relevant to a correct proof.
We hypothesize that explicitly encouraging generated proofs to contain the references will improve correctness, by placing lexical constraints on the reference-titles at decoding time,
[Equation]
where I [ · ] is an indicator function.
To approximate this objective, we generate step-by-step by sampling multiple proof-step candidates, retaining those with high value (reference coverage and log-probability) in a beam, and continuing to the next step, which we call stepwise beam search.
Value function.
The search supports any function of the proof-so-far, v ( y ≤ t )
→ R .
We use a value function that is a weighted combination of constraint satisfaction and log-probability,
[Equation]
) where v constraint ( y ≤ t ) is the number of unique in-context reference-titles in y ≤ t , and v LM ( y ≤ t ) is log p θ ( y ≤ t ) .
We normalize each term by dividing by the maximum absolute value among candidates.
Stepwise beam search.
The procedure generates a proof y =
( y 1 , . . .
, y T ) by iteratively sampling and pruning next-proof-step candidates y t .
Each iteration expands a size- K beam of proofs-so-far,
S t − 1 = { y k<t } Kk =1 , by generating N next-step candidates,
[Equation] [Equation]
where q is a decoding algorithm (e.g. temperature sampling) and ◦ is concatenation.
The next iteration’s beam is formed by selecting the top scoring candidates, S t = arg top - K y ≤ t ∈ S ′ t v α ( y ≤ t ) .
When a proof in the beam terminates, it is not expanded further.
The search ends when the beam consists of K terminated proofs.
The highest value proof is returned as the final output.
Stepwise++.
We add two mechanisms for promoting exploration at each step.
First, we expand each prefix in the beam (Eqn. 6) by sampling with multiple temperatures, { y nt ∼ q τ ( ·| y <t , x , R title ) | τ ∈ { τ i } mi =1 } , where q τ is sampling with temperature τ .
This relaxes the commitment to a single temperature for all proof steps, balancing exploration (higher τ ) with exploitation (lower τ ).
Second, rather than selecting the top-K candidates, we select clusters based on different value weights:
S t = ∪ α ∈{
α j } ℓj =1 top K ′ ( S αt ) , where S αt is the set of candidates scored with v α , and K ′ =
K/ℓ .
This interpolates between selecting steps based on likelihood (low α ) and constraint satisfaction (high α ).
Full proof sampling and greedy decoding.
An alternative is to sample full proofs and select the best one according to the value function.
This can be viewed as expansion (Eqn. 6) done at the full proof, rather than the step level.
Moreover, greedy decoding corresponds to expanding only 1 candidate with temperature → 0 .
We formalize this in §E as a segment-level search that contains stepwise++, full proof sampling, and greedy decoding as special cases.
Proof Evaluation
A proof’s correctness is contingent on a variety of factors, including reasoning with past results, performing symbolic derivations, and altogether providing sufficient evidence that the claim is true.
We design a human-evaluation schema that isolates these aspects at the proof-step level, along with a full-proof summary.
Table 1 summarizes the schema, which we overview below.
References.
First, proofs involve deploying statements from references, such as applying a definition or adapting it to fit the context.
Deployments should be consistent with the reference, e.g. deploying the definition of even integer as ‘...by definition, ∃ k ∈ Z : x = 2 k ...’, rather than ‘...
∃ k ∈
Z : x = 2 k + 1 ’, and are a common source of errors in student proofs [14].
Second, proofs use references as justification for steps of reasoning; for instance, Real Addition is
Commutative provides justification for the statement x + y
=
y
+ x where x, y ∈ R , but not for xy = yx .
This aspect is analogous to using an article to justify a claim (e.g. [29]).
Finally, proofs should not hallucinate references, or ‘beg the question’ by self-referencing.
Equations.
Proofs contain a variety of multi-step derivations, ranging from simple arithmetic to more sophisticated derivations (e.g. see Table 17).
A derivation should start with a valid equation given the surrounding context (e.g. x + x = 2 x in Table 1 versus x + x = 3 x ).
Each subsequent step should be a valid derivation from the previous step, e.g. stating =
(2 k + 6) − 1 after y = 2 k + 5 .
Other reasoning , language , & symbolic errors.
A proof should provide sufficient evidence that a claim is true to a human reader; it should not skip steps.
Proof steps should make progress towards proving the goal; in particular, they should not repeat known conditions in the theorem or conclusions made in a prior step.
Finally, our schema leaves room for any other reasoning errors, as well as symbol errors (e.g. undefined symbols) and language errors (e.g. incomplete statements).
[Table]
Since x is an even integer, ∃ k ∈ Z : x = 2 k + 1 .
Invalid Justification
E ( X 2 ) =
(cid:80) nk =1 k 2 Pr ( X = k )
Power Series for Exponential Function
Hallucinated Ref.
From Power of Number are Irrational , 3 √ 2 is irrational.
Self Loop (Proving Pythagoras’s Theorem:)
From Pythagoras’s Theorem, c 2 = a 2 + b 2 .
Reasoning: Equation
[Table]
∀ x ∈ R , x + x = 3 x .
[Table]
(Since x is an even integer, x + 1 = 2 r + 1 ) =
2( r + 1)
Reasoning
[Table] [Table]
is not
[Table]
Therefore, x 3 ≡ 1 or
[Table]
Repetition (Let △ ABC be a right triangle.)
Then △ ABC is a right triangle.
[Table]
x is an even integer.)
So, x + 1 is an even integer.
Language
Let c = √ a 2 \add b 2 be the ( incomplete statement ; unknown symbol \ add )
Symbolic (Let x ∈ R .)
Let y = x ◦ x − 1 .
( undefined operator ◦ for real numbers )
Usefulness and correctness.
To judge the potential utility of language models as assistive systems in natural mathematics, we measure whether generated next-steps and full proofs are potentially useful hints for proving the theorem on one’s own.
Additionally, we measure a summary judgment of correctness.
Note that an incorrect statement can still be helpful; for instance, it could give a hint for the type of reference to use, derivation to perform, argument to make, etc.
Human evaluation protocol.
We measure these aspects through human annotation at a step-wise and an overall level.
For a step-wise annotation, an annotator is presented with the theorem, proof-so-far, and a generated next-step.
The annotator labels the { 0 , 1 } correctness, usefulness, and presence of fine-grained errors outlined above.
After labeling each step of a proof, the annotator rates the full proof’s overall correctness and usefulness on a 0-5 scale.
A rating of 4 or 5 is needed to be considered as correct, and a rating of 3 or above is needed to be considered as useful.
Automatic metrics: lexical content.
As automatic proxies for quality, we compare each generated proof against its ground-truth counterpart using the sentence-level n -gram matching metric G LEU [28], and following work in knowledge-grounded dialogue [37] we use F1 overlap between generated and ground-truth tokens.
Prior to computing the metrics, we normalize the generated and groundtruth proofs by only keeping the surface form of references, removing formatting characters with a
MediaWiki parser, and collapsing any consecutive whitespace into a single space.
Automatic metrics: knowledge grounding.
We define knowledge grounding as meaning that a generated proof contains the same references as those found in the ground-truth proof.
To measure this, we use precision, recall, and F1-score between the reference sets contained in the generated and ground-truth proofs; i.e. m ( { ˆ r 1 , . . .
, ˆ r ˆ R } , { r ∗ 1 , . . .
, r ∗ R ∗ } ) , where m ( · ) is precision, recall, or F1.
We also use Knowledge Token-F1 (kF1) ([37]), the overlap of the generated proof’s tokens with tokens contained in the references mentioned in the ground-truth proof.
Experiments
We use the training and dev splits of N ATURAL P ROOFS -G EN during fine-tuning, and the core evaluation sets consisting of 100 theorems from the validation set and 100 from the test set for evaluation (see §2).
These theorems were selected by the authors such that by looking at the theorem title each author could recall its content and sketch a proof.
While this may shift the evaluation towards an easier slice of the dataset, it was necessary to make human evaluation at a meaningful scale feasible.
We also use the core sets for explorations and ablations.
We finetune three GPT-3 [5] (Curie) models, using the OpenAI API (see Appendix F for details):
[Table]
Baseline GPT-3.
We finetune a baseline GPT-3 model, p θ ( y | x ) , on theorem-proof examples { ( x , y ) } from the training split.
At test time, we condition the model on a test theorem.
N ATURAL P ROVER R ETRIEVE .
We finetune GPT-3 with retrieved references, p θ ( y | x , ˆ r 1 , . . .
, ˆ r 20 ) .
We use a pretrained joint retrieval model f ( x ) → ( r 1 , . . .
, r |R| ) from [44], which was trained to retrieve an input theorem’s ground truth references.
At test time, the model receives a theorem and the top-20 reference titles that are retrieved given the theorem.
N ATURAL P ROVER .
We finetune GPT-3 with human-provided references, p θ ( y | x , r ∗ 1 , . . .
, r ∗ R y ) , where { r ∗ 1 , . . .
, r ∗ R y } is the set of reference-titles in the ground-truth proof.
We use reference-title conditioned examples (Eqn. 1) and reference-reconstruction (Eqn. 2) on the training split/reference set.
At test time, the model receives a theorem and reference titles from its ground-truth proof.
For next-step suggestion we use the human-provided knowledge model (N ATURAL P ROVER ).
Decoding.
For full proof generation, we use stepwise++ decoding with the provided knowledge model, which we refer to as N ATURAL P ROVER ++ , and otherwise use greedy decoding.
We do not use stepwise constrained decoding with retrieved references since these references introduce noisy constraints, nor for next-step prediction since the algorithm is designed for multi-step proofs.
See §F for additional experimental details.
Human evaluation setup.
To evaluate the proofs generated by N ATURAL P ROVER , we recruited 15 students from the Department of Mathematics and Applied Mathematics at the University of Washington, including undergraduate, masters, and Ph.D. students.
The annotators were trained on how to evaluate proof correctness and compensated according to IRB requirements; see §G.2.
For each task, we first reveal the theorem and its gold proof to the annotator.
If they cannot understand a theorem or its gold proof, they may skip evaluating it.
Otherwise, they may proceed to see the model-generated proof, one step at a time, and annotate each step under the step-wise evaluation schema (outlined in §4).
After all the steps are shown and evaluated, for the full-proof generation task, the annotator is asked to annotate the entire proof under the overall evaluation schema.
5.1 Main Results
Our best method is capable of generating correct and useful proofs.
According to human evaluation results (Table 2), our best method is N ATURAL P ROVER with human-provided references and stepwise++ inference.
32% of the proofs generated by this method are rated as correct and 45% are rated as useful as an aid for human proof writers.
On the per-step level, 35% of the proof steps are correct and 47% are useful .
Taking a more granular view, our best method makes significantly less reference , equation , and other reasoning errors than other baselines.
It makes very few language and symbolic errors , meaning it produces mostly complete, well-formatted mathematical statements, and defines and uses symbols accordingly.
It mostly avoids hallucinating references or creating circular proofs.
On the other hand, despite improving over the other methods, the model often struggles with correctly deploying and using references (23.6% reference error rate), as well as symbolic computations (28.5% equation error rate), especially multi-step derivations (21.9% invalid).
What do the model’s correct proofs look like?
We inspected the proofs labeled as correct and found three main categories: (1) reference-assembly proofs whose correctness is heavily determined by reference statements (e.g. Table 18, Table 20); (2) template-adaptation proofs in which the model adapts the structure and content of a training theorem’s proof to prove the unseen evaluation theorem
[Table]
(e.g. Table 21, Table 22); (3) complex proofs that are not fully determined by reference statements and differ significantly from training proofs (e.g. Figure 1, Table 3).
In terms of techniques, our method demonstrates some ability to produce direct proofs (Table 19), proofs by cases (Table 22), by induction (Table 23), utilize references (Table 20) and do symbolic computations (Table 21).
Vanilla fine-tuned GPT-3 struggles with proof generation.
The vanilla fine-tuned GPT-3 model yielded fewer useful and correct proofs, with more reference-based and other reasoning errors than all three knowledge-grounded settings.
The model showed severe reference hallucination (18%) and repetition (23%).
It also makes significantly more reasoning errors related to reference usage.
Language and symbolic error rates roughly stay the same.
Overall, naively fine-tuning GPT-3 on theorem-proof examples alone is suboptimal for proof generation.
Human-provided knowledge improves proof generation.
Grounding the generations with humanprovided references significantly raises correctness and usefulness of the proofs in both fullproof and per-step evaluation.
It most substantially reduces reference errors , especially invalid deployments and hallucinated references.
For example, Table 3 shows the model grounding a proof with information from the theorem Singleton Point is Isolated and the definition of Dense-in-itself, in contrast to the vanilla GPT-3 model which hallucinates references.
Retrieved knowledge also improves proof generation.
Retrieved knowledge also turns out to be very helpful, and even comparable to human-provided knowledge in some metrics.
Although the retrieval model is far from perfect, the proof generation model is capable of narrowing down the retrieved reference titles provided in its context, assembling proofs that are useful and correct more often than the no-knowledge model.
Qualitatively, we found examples where grounding in retrieved references eliminates repetition, enables multi-step derivations justified by references (Table 21), and assembles references into a correct proof (Table 20).
This paves a promising path towards fully automated mathematical proof generation in natural mathematical language.
[Table]
the constrained objective better than greedy decoding, resulting in both lower perplexity and better reference coverage, regardless of whether knowledge is provided in-context.
Constrained decoding further improves proof generation.
Table 4 confirms that stepwise++ decoding approximates the constrained objective (Eqn. 4) better than greedy search, yielding proofs with lower perplexity and higher constraint satisfaction (Ref-F1).
This translates to generations that are correct and useful more often according to the annotators.
Intuitively, the constraints encourage the model to include references that help prove the claim (e.g. Table 18).
Next-step suggestion.
The next-step suggestion task characterizes a model’s performance on making a single proof step given a correct proof-so-far.
In Table 2 we use the provided-knowledge model with greedy decoding for next-step suggestion, and find that reasoning errors decrease and per-step usefulness and correctness improve compared to the full proof setting, with 51% of the proof steps rated as useful and 43% correct.
Although we used a single suggestion in our human evaluation study, in Table 5 we simulate a user choosing from among multiple suggestions by sampling 10 next-steps from our model and computing automatic metrics on the sample with the best sum of metrics.
Using 10 samples instead of greedily decoding a single sequence substantially improves each metric, suggesting that utility might be increased further by presenting multiple suggestions.
[Table] [Table]
[Table] [Table]
Sampling 10 suggestions improves over a single greedy suggestion.
How good are Automatic Metrics?
We study how well the automatic lexical and grounding metrics introduced in (§4) can reflect the real quality of proofs, as a guide for using them as a proxy evaluation protocol for N ATURAL P ROOFS -G EN .
We compute the Pearson correlation coefficient between each pair of human and automatic metrics, with data from the four experiment settings for full-proof generation.
Results are shown in the lower part of Table 6, with error metrics negated, meaning positive correlation is desired.
The lexical and grounding metrics positively correlate with full proof correctness and usefulness ( ≥ 0.8).
At the step-level, the metrics show (i) high correlation with step-level correctness and language errors ; (ii) varied, but positive, correlations with aggregate reasoning errors; (iii) negative correlation with symbolic errors (though symbolic errors are relatively low for all models).
The results suggest that optimizing for automatic metrics may be a viable strategy, albeit without guarantees on how finer-grained reasoning aspects vary across proofs.
5.2 Ablations and error analysis.
[Table]
Reference reconstruction.
We fine-tune an additional GPTmodel that is provided with in-context reference titles, but without reference reconstruction.
As seen in Table 7, reference reconstruction improves content and reference usage.
Constrained decoding.
First, Table 9 compares the step-level search in stepwise++ with searching at the full-proof level through sampling multiple proofs and selecting the best with the N ATURAL P ROVER value function ( rerank (n) ).
Reranking 60 samples matches the cost of stepwise++ in terms of number of decoded tokens.
Full-proof reranking yields the best Gleu, though with lower reference-F1.
Second, Table 8 shows that the expansion and selection mechanisms together result in the best reference matching, while holding Gleu at a similar level.
Finally, Table 13 shows that both terms in the N ATURAL P ROVER value function αv constraints + (1 − α ) v LM are needed: increasing the constraint weight α increases reference-matching, with a tradeoff in Gleu at high values.
Language model comparison.
Table 10 varies the language model used to parameterize N ATURAL P ROVER .
The content and reference usage metrics improve with larger models.
Separately, we find that increasing inference-time compute closes the gap in reference-matching between GPTand the larger GPT-3 model (Table 11): sampling 10 full-proofs from GPT-2 and selecting the best using the N ATURAL P ROVER value function achieves the same reference-F1 as GPT-3 with a single greedily-decoded proof.
However, Gleu remains much higher with the larger GPT-3 model.
Challenge: Reasoning with references.
Although reference reasoning errors were decreased through knowledge-grounding and constrained decoding, N ATURAL P ROVER still commits a reference error on 23.6% of test steps (27% dev), with 15% of steps containing invalid deployments and 10% invalid justifications.
For next-step prediction, the reference error rate remains nontrivial (19.7% test, 13% dev).
, meaning that the model can struggle to correctly deploy references or use them as justification even in the absence of compounding errors from previous steps.
Table 15 shows example invalid deployments and justifications; the errors are at times subtle, and require reasoning about the theorem statement, reference content, and proof context.
Challenge: Equations and derivations.
N ATURAL P ROVER commits an equation-related error on 28.5% of test steps (22.8% dev), including invalid equations (9.4%) and derivations (21.9%).
Though an improvement over vanilla fine-tuned GPT-3 (32.5%), the errors occur frequently and remain high for next-step prediction (26%).
Table 17 shows representative errors, which range from simple ‘commonsense’ mistakes (e.g. 24 = 2 3 ) to making invalid steps with false justification within more sophisticated multi-step proofs.
Investigating the role of pretraining, in-context techniques [30], and autoformalization [38] is interesting future work.
Challenge: Proof length.
Although N ATURALP ROVER demonstrates some ability to write long proofs (e.g. Table 23), the 42% next-step correctness suggests that compounding errors are likely as proof length increases.
Indeed, our best model’s full-proof correctness is 48% on 1-4 step proofs ( n = 102 ), decreasing to 15.6% on proofs with 5 or more steps ( n = 64 ), with lower per-step usefulness and correctness at later steps (Figure 2).
Our findings are analogous to recent work on language modeling for formal theorem proving [31], where current models are typically limited to chaining 2 or 3 non-trivial steps of mathematical reasoning.
Related Work
Language models have been used for interactive theorem proving in formalized mathematics, in which proofs steps are verifiable and written in a programming language [32, 31, 19, 40].
Previous work on informal theorem proving focuses on premise retrieval [16, 17, 44, 20], or informal-to-formal translation [42], which differs from generating next-steps or full proofs.
Outside of theorem proving, language models are used for problem solving [36, 21, 7, 43], math-related program synthesis [24, 2, 13] and competitive programming [25].
These settings generate executable programs or a numerical answer, which differ from our theorem proving setting, where the goal is generating arguments on many topics in natural mathematical language.
See Appendix C for more related work.
Conclusion
We described N ATURAL P ROVER , a knowledge-grounded language model that generates mathematical proofs by conditioning on background theorems and definitions, and optionally enforces their presence with constrained decoding.
Our system improves the quality of next-step suggestions and generated proofs over fine-tuned GPT-3, demonstrating an ability to correctly prove theorems and provide useful suggestions to human proof writers.
Acknowledgments and Disclosure of Funding
This work was funded in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) (funding reference number 401233309), DARPA MCS program through NIWC Pacific (N66001-19-2-4031), and the Allen Institute for AI.
We also thank Google Cloud Compute, as well as OpenAI.
The authors would like to thank Alisa Liu, Julian Michael, Yuren (Rock) Pang and Kaiming Cheng for dogfooding and providing valuable feedback to our human evaluation system.
We would also like to thank James McGivern for developing an interactive demo for NaturalProver
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See §5.2.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See §H. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We will release our code in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See §D and §F. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See §F. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] See §D. (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See §G.2 and supplementary materials.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See §G.2.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See §G.2.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[Yes] See §G.2 and supplementary materials.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[Yes] See §G.2.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[Yes] See §G.2.
A Additional Results A.1 Additional ablations
Table 10 shows automatic metrics with various language models used to parameterize N ATURALP ROVER .
Table 11 shows results with the 774M parameter GPT-2 model with greedy decoding, and full-proof sampling & reranking with 5 and 10 samples, compared to the 13B parameter GPT-3 with greedy decoding.
We use τ = 0 .
3 and α = 0 .
75 based on our full-proof sampling experiments with GPT-3.
Table 13 varies the value function parameter α (core dev set).
We use full-proof sampling since stepwise++ uses multiple values of α in its selection.
[Table] [Table]
[Table] [Table]
The baseline search corresponds to using stepwise decoding
LM
Constrained stepwise++ decoding substantially improves grounding metrics compared to stochastic beam search, while keeping the lexical content metrics at a similar level.
Core validation set.
A.2 Multiple next-step suggestions
Table 14 shows next-step suggestion results with 10 sampled suggestions versus greedy decoding.
Lexical
Grounding
Decoding
[Table]
B Qualitative Examples B.1 Error Analysis B.1.1 Reference errors
Theorem
Geometric Mean of two Positive Real Numbers is Between them Let a,b ∈ R be real numbers such that 0 < a < b .
Let G ( a,b ) denote the geometric mean of a and b .
Then: a < G ( a,b ) <
b
N ATURAL P ROVER ++ Proof: correct deploy By definition of geometric mean:
G ( a,b ) =
a 2 + b 2✗ ✗ ...
Gold Proof
Proof: By definition of geometric mean:
G ( a,b ) := √ ab where √ ab specifically denotes the positive square root of a and b . ...
Reference
Geometric Mean Let x 1 ,x 2 ,...,x n ∈ R > 0 be (strictly) positive real numbers.
The geometric mean of x 1 ,x
[Table]
,...,x n is defined as:
G n := ( (cid:81) nk =1 x k ) 1 /n .
Theorem
Power Set of Transitive Set is Transitive Let S be a transitive set.
Then its power set P ( S ) is also a transitive set.
N ATURAL P ROVER ++ Proof:
[Table] [Table]
P ( S ) ✓
From
[Table]
✗ ✗ ...
[Table]
Reference
[Table] [Table]
[Figure]
B.1.2 Equation errors
Theorem
Sum of Squares of Divisors of 24 and 26 are Equal
The sum of the squares of the divisors of 24 equals the sum of the squares of the divisors of 26 : σ 2 (24)
=
σ 2 (26) where σ α denotes the divisor function.
Proof: correct eqn.
N ATURAL P ROVER ++ 24 = 2✗ ✗ 26 = 2✗ ✗ ⇝ σ 2 (24) =
σ 2 (26) ✗ ✗ ■
Theorem
Trace of Sum of Matrices is Sum of Traces Let A =
[ a ] n and B =
[ b ] n be square matrices of order n .
Let A + B denote the matrix entrywise sum of A and B .
Then: tr ( A + B ) = tr ( A ) + tr ( B ) where tr ( A ) denotes the trace of A .
Proof: correct eqn.
ref.
N ATURAL P ROVER ++ tr ( A + B ) =
(cid:80) C =
A + B tr ( C )
Trace of Matrix ✗ ✗ ✗ =
(cid:80) C
=
A tr ( C ) + (cid:80) C
= B tr ( C ) Sum of Summations equals Summation of Sum ✗ ✗ ✗ =
tr ( A ) + tr ( B )
Trace of Matrix ✗ ✗ ✗ ■
Theorem
Moment Generating Function of Discrete Uniform Distribution Let X beadiscrete random variablewithadiscrete uniform distribution with parameter n for some n ∈ N .
Then the moment generating function M X of X is given by:
M
[Equation]
nt ) n (1 − e t )
Proof: correct eqn.
ref.
N ATURAL P ROVER ++ From the definition of the discrete uniform distribution ,
X has probability mass function: Pr( X = k )
=
n n + k ✗ ✗ ✗
From the definition of a moment generating function:
M X ( t ) =
E e tX =
(cid:80) nk =
Pr( X = k ) e tk ✓
So: M X ( t ) =
(cid:80) nk =
Pr( X = k ) e tk ✓ = nn +
[Table]
✗ ✗ ■
B.2 Examples
Theorem
Metric Space is Perfectly Normal
Gold Proof Let M =
( A,d ) be a metric space.
By definition, a topological space is perfectly normal space
iff it is: Then M is a perfectly normal space.
perfectly T 4 space and T 1 (Fréchet) space.
We have that:
Metric Space is Perfectly T 4 Metric Space is T 2 (Hausdorff)
T 2 (Hausdorff) Space is a T 1 (Frechet) Space.
■
N ATURAL P ROVER ++ From:Metric Space is
[Table]
N ATURAL P ROVER (without stepwise
[Table]
+) From:Metric Space is Perfectly
[Table]
T
[Table]
it follows
[Table]
is
[Table]
topological
[Table]
which is perfectly
[Table]
Metric Space is T
[Table]
it follows
[Table]
M is a
[Table]
normal
[Table] [Table]
[Equation]
[Table] [Table]
[Table]
Theorem Content:
Gold Proof
N ATURAL P ROVER
C Expanded Related Work
Formalized mathematics with neural language models.
A large portion of work on machine learning for mathematics focuses on formalized mathematics.
Language models have been used for interactive theorem proving, including in GPT- f [32, 31], PACT [19], and in [40].
In these settings proof steps are expressed in a programming language (e.g. Lean [10]) and there is access to a verifier, which differs from our setting of theorem proving in natural mathematical language.
Informal mathematics with neural language models.
Previous work on theorem proving in natural mathematical language focuses on retrieving relevant premises (e.g. theorems, definitions) [16, 17, 44, 20], or informal-to-formal translation [42], which differ from our setting of generating next-steps or full proofs.
Outside of theorem proving, various works use sequence models for problem solving, including benchmarking language models on arithmetic [36] or competition problems [21], symbolic mathematics [24, 45], augmenting LMs with verifiers [7] or in-context rationales [43] for math word problems, or using language models for math-related program synthesis [2, 13] and competitive programming [25].
These settings focus on generating executable programs or a numerical answer, which differ from our theorem proving setting, where the goal is to generate sound and convincing arguments on a range of topics in natural mathematical language.
Related areas in NLP.
Systematic reasoning in natural language (outside of math) has been studied with synthetic proofs [35, 39], single-step deductions [4], or entailment trees [8], which differ from proving real-world mathematical theorems.
Augmenting LMs with knowledge reduces hallucinations in dialogue [37] which has an analogous step-wise structure, while [29] use references within longform answers; these and related NLP findings differ from improving the utility of mathematical proofs.
Lexically-constrained decoding algorithms include variants of (token-level) beam search (e.g. [1, 22, 27, 26]) which assume access to per-token logits, and gradient-based decoding [33]; our segment-level decoding only assumes a sampler that returns text and its log-probability (e.g. the
GPT-3 API).
D Dataset Details
We provide an overview of N ATURAL P ROOFS and its ProofWiki domain from which we build
N ATURAL P ROOFS -G EN .
Refer to [44] for further details about N ATURAL P ROOFS .
Our dataset is derived from N ATURAL P ROOFS , a multi-domain corpus of theorem statements, proofs, definitions, and additional pages (e.g. axioms, corollaries) in natural mathematical language.
We use the ProofWiki 2 domain, which provides broad-coverage of many subject areas (e.g. Set Theory,
Analysis) sourced from ProofWiki, an online compendium of community-contributed mathematical proofs.
P ROOF W IKI contains ∼ 20k theorems, ∼ 20k proofs, ∼ 12k definitions, and ∼ 1k additional pages (e.g. axioms, corollaries).
The set of all ∼ 33k theorems, definitions, and additional pages form the reference set R .
Finally, ∼ 14.5k of the theorems x are paired with at least one proof y to form examples D = { ( x , y ) i } Ni =1 .
[44] split the reference sets and examples into training, validation, and test splits to ensure that no theorem in the validation or test splits was mentioned in the training split.
E Segment-level Constrained Decoding
In this section we present a generic segment-level decoding algorithm that contains stepwise++, full-proof sampling, and greedy decoding as special cases.
We generate a multi-step proof using a value function v ( · ) that measures language quality and constraint satisfaction.
Search can be done at the step-level, in which candidate next-steps are generated and high-value steps are retained in a beam, or at the proof-level, in which multiple proofs are generated and the highest-value proof is selected.
We formalize these into a generic segment-level search , where a segment s t is either a proof-step y t or a full proof y .
The search iteratively builds a multi-step proof y =
( y 1 , . . .
, y T ) by expanding , scoring , and selecting a set of candidate segments :
• Expand : S t − 1 →
• Score : ( s ≤ t , v ) → R scores a candidate using a value function, v ( s ≤ t )
→ R .
• Select : S ′ t → S t prunes candidates S ′ t into segments S t used in the next iteration.
Value function.
We score candidates based on constraint satisfaction and language quality,
[Equation]
where v constraint ( y ≤ t ) is the number of unique in-context reference-titles in s ≤ t , and v LM ( s ≤ t ) is log p θ ( s ≤ t ) .
We normalize each term by dividing by the maximum absolute value among candidates.
Greedy search.
This baseline search defines a segment as a full proof, meaning s 0 is an empty sequence and s 1 is a proof y .
Expand samples one segment candidate with temperature 0.
Score and select are trivial since there is only one candidate.
Greedy search costs T steps of tokens.
Sample-and-rerank.
In this search, a segment is again full proof, but expand samples N candidates, S ′
= { y n ∼ q ( ·| x ) } Nn =1 , where q is a decoding algorithm (e.g. temperature sampling).
Select takes the top scoring candidate, y = arg max y n ∈ S ′ 1 v ( y n ) .
The cost is NT steps of tokens.
Step-wise stochastic beam search.
This search generates by iteratively sampling and re-ranking next-step candidates.
In this case, a segment is a proof step, y t , and each iteration starts with a beam of proofs-so-far, S t −
= { y k <t } Kk =1 , where K is the beam size.
Expand samples N next-step candidates for each proof-so-far in the beam,
[Equation]
where q is a decoding algorithm (e.g. temperature sampling) and ◦ is concatenation.
Select forms the next beam using the top- K scoring candidates,
[Equation]
When a proof in the beam terminates, it is not expanded further.
The search ends when the beam consists of K terminated proofs.
The highest scoring proof is returned as the final output.
The cost is
NTK steps of tokens.
Stepwise++.
At certain proof steps it is important to enumerate and explore options, while at others (e.g. derivations) a single highly probable prediction is better.
To this end, we expand by sampling with multiple temperatures, meaning that we expand each prefix y <t in (6) using:
[Equation]
where q τ is sampling with temperature τ .
This relaxes the commitment to a single temperature for all proof steps, intuitively balancing exploration (higher τ ) with exploitation (lower τ ).
Second, during the search we want to balance selecting proof steps that satisfy constraints and proof steps with high log-probability.
To this end, we select clusters with different value weights,
[Equation]
where S α means the set of candidates scored with v = αv constraint + (1 − α ) v LM , and K ′ =
K/ℓ .
This interpolates between selecting steps with good language score ( α small), constraint score ( α large), and balance ( α : 0.5).
F Implementation Details and Experimental Setup
Data preprocessing.
We automatically infer the boundaries of proof steps within the raw proof contents, and merge contiguous lines into atomic proof steps when appropriate.
Steps are separated by the \n token ( \\n in Python string), and lines within a step are separated by the newline token ( \n in Python string).
Implementation details.
All N ATURAL P ROVER and GPT-3 models are fine-tuned with the OpenAI
API 3 for 4 epochs with a batch size of 64.
Other models (GPT-2/J/Neo) are trained on one Quadro
RTX 8000 GPU.
During inference, the prompt (up to <proof> ) is truncated to 1024 tokens.
For full proof generation, we allow a maximum of 1020 generated tokens.
For next-step suggestion, we truncate the proof-so-far to 900 tokens, and allow a maximum of 120 generated tokens per step.
Additional model details.
All GPT-3 models are fine-tuned instances of the Curie engine, the second largest model available through the OpenAI API at the time of writing.
The model’s performance on the EleutherAI evaluation harness 5 is between the 6.7B and 13B variants of the autoregressive transformer language model GPT-3 from [5], 6 though further details of the Curie model are not publicly available.
Separately, we fine-tune GPT-J 6B, 7 a publicly available autoregressive transformer language model trained on the Pile [18], GPT-2 [34], an autoregressive transformer language model trained on scraped web documents, and GPT-Neo-125M, 8 a GPT-2 like causal language model trained on the Pile.
Our retrieval model is the joint retrieval model from [44] trained for reference retrieval on ProofWiki using the same dataset splits as NaturalProver.
We use the publicly-available pretrained model from the GitHub repository of [44] and do not update the model further.
We use the model to retrieve the top-20 references for each input theorem.
Stepwise++ decoding.
For expansion with multiple temperatures, we use N = 10 candidates sampled with ( n, τ ) ∈ { (1 , 0 . 0) , (3 , 0 . 3) , (3 , 0 . 5) , (3 , 0 . 7) } .
We also tried including τ = 1 . 0 which resulted in very poor G LEU , and {(1,0.0), (5,0.3), (4,0.5)}.
For selection, we use a beam size K = 9 , and three equally-sized clusters formed with α ∈ { 0 . 1 , 0 . 5 , 1 . 0 } .
We also tried { 0 . 5 , 0 .
We use α = 0 .
75 to pick select the final sequence, based on our ablation with full-proof sampling.
Full proof sampling.
We use temperature τ = 0 . 3 , selected based on a search over τ ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 .
7 } using GLEU plus Ref-F1 on the core dev set.
G
Additional Evaluation Details G.1 Full Evaluation Schema
Table 24 shows the full schema of human evaluation.
The overall correctness and usefulness are rated on a 0-5 scale.
The step-wise correctness and usefulness are yes/no questions, while the error types ask for a binary indicator for the existence of each error type.
G.2 Additional Human Evaluation Details
Process.
The authors conducted and moderated group sessions with the annotators.
Each session consisted of 30-minutes of training and a 1-hour working/Q&A period.
After attending the session, annotators could continue working on their assigned tasks for two weeks.
Each annotator was assigned 25 theorems (with 5 proofs per theorem, equaling 125 total tasks) and asked to complete as many tasks as they would like.
The evaluation guideline that the annotators referenced to can be found in the supplementary materials.
The pre-recorded training video is available at https: //drive.google.com/file/d/1TRS5XRf_coLEkC4lqaizaqSwHHgBPrG2 .
Interface.
We developed an interface that displays theorems and proofs in a rendered, humanreadable format and collects annotations.
The interface is built on MediaWiki 9 , which also powers the ProofWiki website 10 .
We also developed a web console that helps human annotators navigate annotation tasks and track progress.
Figure 3 shows screenshots of the interface.
Payment.
Human annotators are paid based on the number of tasks they complete.
Each task is worth ($1 .
+ #steps × $0 . 4) .
We pay each annotator an additional $40 for attending the group session.
Annotators are guaranteed a minimal rate of $20/hour.
The human evaluation costs approximately $5,000.
4 https://beta.openai.com/docs/guides/fine-tuninghttps
github.com/EleutherAI/lm-evaluation-harnesshttps://blog.eleuther.ai/gpt3-model-sizes/https
huggingface.co/EleutherAI/gpt-j-6Bhttps github.com/EleutherAI
[Table]
◦ 0:
[Table]
proof
[Table]
missing.
◦ 1: The proof makes no sense or is unrelated to the problem statement.
◦ 2: The proof contains serious logical flaws and lacks adequate justification or explanation.
◦ 3: The proof has some gaps in reasoning.
◦ 4: The proof is correct or nearly correct and logically coherent.
[Table]
5: The proof is correct and flows logically.
[Table]
Even if the proof is not perfect, would it be useful to you if you were to prove this theorem?
[Table]
[Table]
proof
[Table]
as
[Table]
solution.
[Table]
Is this step correct? ◦
Yes ◦
No (
[Table]
this if you identified any error in previous questions)
[Table]
Cannot
[Table] [Table]
[Table]
Could this step be a helpful hint for proving the theorem by myself? ◦
Yes ◦
No
Reasoning
[Table]
Reference
[Table]
A statement deployed from a reference is not consistent with the reference.
[Table]
A reference is used as invalid justification for a statement.
[Table]
Ref
[Table]
A reference that does not exist is used.
[Table]
The step refers to the theorem itself.
[Table]
Equation
[Table]
A standalone equation or initial equation in a derivation is invalid.
[Table]
An equation in a derivation does not follow from the preceding steps.
[Table]
Other
[Table]
The step assumes unproven statements, or skips non-trivial steps.
[Table]
The step is merely a repetition of known things.
Invalid (Other)
[Table]
[Table] [Table] [Table]
[Table] [Table]
Ethics review. The human evaluation study is approved by University of Washington under IRB STUDY00014751. Consent was obtained from each human annotator by signing a consent form via
DocuSign prior to the beginning of study.
The IRB approval letter and a template of the consent form can be found in the supplementary materials.
Minimal personally identifiable information (PII) was collected [Table] and removed prior to any data analysis.
G.3 Full results
Table 25 shows the full results of human evaluation, including the error rates of fine-grained error types.
G.4 Analyzing the Annotators
Inter-annotator agreement.
We compute inter-annotator agreement using proofs in the core dev set that get an evaluation from two or more annotators.
Overall, the annotators achieved fair agreement (Fleiss kappa κ = 0 . 24 ).
The level of agreement for each evaluation question is shown in Figure 4.
Fair to moderate agreement is reached for identifying coarse-grained error types, while the high-level questions (i.e. correctness , usefulness ) have relatively low agreement.
Source diversity.
Figure 5 shows the largest proportion of evaluations covered by a fixed number of annotators.
The top-1 annotator contributes 20% of the total evaluations when counting by proofs and 18% when counting by steps.
50% of the total evaluations is covered by roughly the top 3 orannotators.
Therefore, our human evaluation results have good source diversity and do not heavily depend on a single annotator’s opinion.
H Ethical Considerations
Our system may produce proofs of mathematical theorems that are fallacious or misleading, which may have negative impact if deployed in real educational environments.
We kindly remind potential users that our system and models are experimental, and their outputs should be interpreted critically.