Autoformalization with Large Language Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Autoformalization is the process of automatically translating from natural languagemathematics to formal speciﬁcations and proofs.
A successful autoformalizationsystem could advance the ﬁelds of formal veriﬁcation, program synthesis andartiﬁcial intelligence.
While the long-term goal of autoformalization seemedelusive for a long time, we show large language models provide new prospectstowards this goal.
We make the surprising observation that LLMs can correctlytranslate a signiﬁcant portion (25.3%) of mathematical competition problemsperfectly to formal speciﬁcations in Isabelle/HOL.
We demonstrate the usefulnessof this process by improving a previously introduced neural theorem prover viatraining on these autoformalized theorems.
Our methodology results in a newstate-of-the-art result on the MiniF2F theorem proving benchmark, improving theproof rate from 29 .
6% to 35 .
Introduction
Autoformalization refers to the task of automatically translating from natural language mathematicsto a formal language [46, 42].
The implication of a successful autoformalization tool is huge inboth practical and philosophical terms.
It would reduce the currently excessive cost of formalizationefforts [27], and in the long-term it could connect the various research ﬁelds that automate aspects ofmathematical reasoning, such as automated theorem proving and computer algebra, to the vast bodyof mathematical knowledge exclusively written up in natural language.
Moreover, autoformalizationwould be a true testament to machine understanding, grasping both the fuzziness of natural languageand the preciseness of formal language.
Recent advances in large language models [8, 10] showed promising capabilities of understandingformal languages [9, 32].
However, the existing successes are limited to formal languages wherethere exists a large body of corpus on the web (e.g., Python language).
Formal mathematics data isvery scarce.
For example, one of the largest formal mathematics libraries, the Archive of Formal
Proofs, is only 180MB in size, that is less than 0 .
18% of the training data for the large languagemodel Codex [9].
Moreover, unlike in the case of commonly used programming languages, wherenatural language docstrings are broadly available, there is almost zero aligned data between naturallanguage and formal mathematics.
Therefore, it is unclear the recent successes can directly contributeto the development of autoformalization.
In this work, we explore the prospects of autoformalization with large language models.
To oursurprise, we ﬁnd that large language models already have a decent capability of formalizing naturallanguage mathematics in an interactive theorem prover.
See Figure 1 for a perfect autoformalizationexample.
The model not only translates into syntactically correct Isabelle code, but also graspsthe non-trivial reasoning in natural language.
We randomly pick 150 formalizations and manuallyevaluate their correctness.
Among them, LLMs are capable of producing 38 perfect formalizations!
As an application, we further demonstrate that autoformalization can provide useful training datafor neural theorem provers.
We use autoformalized statements as targets for proof search with a
Case Study 1 Question: "Prove that there is no function f from the set of non-negative integers into itself such that f ( f ( n ))
= n + 1987 for every n ."
Codex Output: theorem fixes f :: "nat \<Rightarrow> nat" assumes "\<forall> n. f (f n) = n + 1987" shows False neural theorem prover for it found, its success rate on the benchmark [52] increases signiﬁcantly, achieving a new -of the of 35.2%
Related Work
Early applications of machine learning in theorem proving include the works by Schulz [40] and Urban[43], and later, directly guiding interactive proof assistants using machine learning techniques [15].
The revolution of deep learning then kicked off a new wave of interest in the topic starting with
DeepMath [1, 33].
Several approaches have been suggested to address data scarcity: Imitation-free reinforcementlearning was used to avoid the need for training on human proofs [31, 6, 15, 49].
Also, hindsightexperience replay [2] was used to generate additional training data [5].
Hahn et al.
[19], Schmitt et al.[39], Kreber & Hahn [29] and Wu et al.
[50] have shown that training on synthetic formulas can besuccessful for temporal logics and inequalities.
Rabe et al.
[37] masked out different subexpressionsfrom formal mathematical statements and generated 100 training examples for each source statement.
Skip-tree data can also be used to improve the performance of neural theorem provers [22].
Wang et al.
[46] explored the use of supervised and unsupervised translation techniques for autoformalization.
Supervised translation yielded interesting results, but relied on synthetic (natural-looking)data that was generated by the Mizar theorem prover, while we rely on models trained via selfsupervised language modeling, not trained for this particular purpose.
Background
Formal Mathematics
A few important and complex results of mathematics and computer sciencehave been formalized manually using interactive theorem provers , such as the four color theorem [16],the Kepler conjecture [20], the odd-order theorem [17] and the veriﬁcation of a microkernel [27].
This gives us almost complete certainty about the correctness of proofs, which can be of great value toresolve doubt about the correctness of complicated mathematical proofs or proving certain propertiesof software used in safety-critical applications, such as aircraft components [28].
These projects relied on interactive theorem provers, such as Isabelle [48], Coq [12], HOL Light [23],and Lean [13], which are essentially programming languages that enable users to enter their statementsand proofs in a formal language, and which can then be checked automatically for correctness.
Interactive theorem provers offer a limited amount of automation, but projects that formalize complexproblems typically span many years of tedious work by specialists.
Only in narrow domains like chipdesign and the veriﬁcation of drivers in operating systems has the automation of logic made sufﬁcientprogress to ﬁnd commercial applications.
Progress in autoformalization and the automation of proofs might eventually make mathematics auniversally available tool and enable a paradigm shift in science and the development of (safetycritical) software.
Our interest in formalizing mathematics, however, has an additional aspect.
Webelieve that autoformalization will serve a dual purpose and will not only accelerate the development of tools for mathematical reasoning, but also provide a means to ground machine learning systems,enabling a positive feedback loop between machine learning and formal systems (cf.
Large Language Models Our work relies heavily on large language models (LLMs), in particularon PaLM [10] and Codex [9].
The training goal of these models is to predict the next word givensome preﬁx.
This allows us to train these models on arbitrary text, which is available in vast quantities.
After training the models on hundreds of billions of words (cf.
[25]), they are often able to generatehigh-quality text.
We can also give these models an arbitrary preﬁx (the prompt ) that they arethen supposed to continue, which gives us some control over what they generate.
This has beendemonstrated with news articles, conversations, summaries, jokes, and poems.
LLMs have also beenevaluated on natural language word problems on datasets such as GSM8K [11] and MATH [24], andhave been shown to make progress on these benchmarks with increasing scale [10].
In-context Learning Large language models have shown a remarkable ability to learn patternsand tasks within the current input (context) that they are given [8]: this is called in-context learningor few-shot learning .
For example, if we prompt a language model with a few pairs of English andmatching French sentences, and end with a new English sentence, then the language model is verylikely to pick up on the translation task and attempt a translation of the last English sentence.
Thisobservation has been used, for example, to achieve strong translation performance without access tolarge corpora of matching sentence pairs [21].
This allows us to specify the task of autoformalization simply by giving a couple of example formalizations.
In Section 4 we will detail how exactly we use in-context learning for autoformalization.
Autoformalization for Mathematical Competition Problems
Inspired by the success of LLMs for synthesizing computer code by co-training on both naturallanguage and code on web-scale data, we explore the capabilities of LLMs to turn natural languagemathematics into formalized theorems for the interactive theorem prover Isabelle.
This can be seenas a machine translation task (cf.
[47]) in which the input language is English and output language isformal code used by the interactive proof assistant Isabelle [48].
We ﬁrst study autoformalization in a constrained setting – formalizing mathematical competitionproblem statements.
This setting has the advantage that most of the required background theory anddeﬁnition has been formalized in the current libraries of Isabelle, so that formalizations are oftenpossible without introducing additional deﬁnitions.
We start assessing LLMs’ abilities to do autoformalization with a case study.
We manually picktwo interesting natural language mathematical statements, and prompt PaLM models of variousscales [10] as well as Codex [9] to translate them into a formal statement in Isabelle.
Next, westudy a dataset in which we have human ground truth formalizations.
The dataset is a subset of theminiF2F [24] dataset consisting of 140 algebra problems and 120 number theory problems.
Usinghuman formalizations as the reference, we compute the BLEU scores of the formalizations producedby several LLMs.
Lastly, we perform human evaluations on failure cases in autoformalization on 150problems.
Note that many mathematical competition statements are often of the form in which one asks to ﬁnd the answer to a certain problem, instead of proving a given proposition.
However, formal mathematical statements are in the form of propositions, instead of questions.
To transform a question into a proposition, we append the ﬁnal answer after the question: $Problem_Statement The ﬁnal answer is $Answer .
The format of the prompt we use to do autoformalization is:
Natural language version: $Natural_Language_Statement .
Translate the natural language version to an Isabelle version:
4.1 Mathematical Competition Datasets
MATH [24] contains in total 12,500 (7,500 training and 5,000 test) middle school and highschool mathematical competition problems.
Problems are taken from past mathematical com- petitions, including AMC 10, AMC 12, AIME, and more, and many can be found at http://aops.com/community/c3158_usa_contests .
The dataset contains seven categories: algebra , pre-algebra , intermediate algebra , number_theory , precalculus , probability ,geometry .
Problem statements are written in LaTeX.
MiniF2F [52] is a recently introduced benchmark containing 488 mathematical competition statements manually formalized by humans in three different formal languages.
Its goal is to compareand benchmark methods across different theorem provers for machine learning research.
Some ofthese problems come from the valid and test set of MATH algebra and number_theory , and otherscome from previous International Mathematical Olympiad competitions or AoPS 1 .
Note that the
Isabelle formalizations of the miniF2F benchmark were committed to the repository during March,2022.
According to the public information of the training data, we think it is highly unlikely theseformalizations were included in the pre-training corpus.
4.2 Case Studies
Experimental setup For all our experiments, we use the standard greedy decoding (i.e., temperature0, p = 1 ) to obtain the autoformalizations.
We randomly select two mathematical statements forconstructing the prompt, which we provide in Appendix ?? .
That is, no prompt engineering / tuningis performed when constructing the prompt.
The natural language problem statements used in thecase studies are taken from the miniF2F dataset.
In the case studies below, we highlight the output oflanguage models in red to distinguish it from the prompt.
Case Study 1 (Figure 1)
We study the example shown in Figure 1, in which we ask LLMs toautoformalize an International Mathematical Olympiad problem 2 in natural language.
Surprisingly,
Codex is able to autoformalize the natural language statement as an Isabelle theorem perfectly, withoutput given.
This is surprising for the following reasons.
First of all, the amount of Isabelle code is very scarce on the internet.
The entire AFP library, thelargest formal library that contains most of Isabelle proofs, is only 180MB in size.
Even assumingthat all of this data was included in the training of Codex, this makes at most 0 .
18% of the pretrainingdata on which Codex was trained.
The fact that the model can write syntactically correct Isabellecode at all is already fascinating.
Second, there is almost zero aligned data from natural language to Isabelle on the web.
While some
Isabelle ﬁles have comments, they typically only give a very high level description of what the theorybeing formalized is about.
So either LLMs are able to transfer knowledge quite successfully betweennatural language and formal mathematics, or the task was learned mostly via few-shot learning.
Last but not least, the model is capable of understanding and formalizing nontrivial reasoning.
First,the model is able to formalize the non-existence statement via proof-by-contradiction.
To formalize“there is no function f ...”, it assumes there is such a function, and aims to prove “False”.
Second,the model understands what it means by the phrase “to itself”, and correctly infers the domain offunction: f :: "nat \<Rightarrow> nat" .
On the other hand, PaLM made some syntactic mistakes while getting most of the structure of theproof correctly, with outputs shown in Appendix ?? .
Case Study 2 (Figure 2)
In the next example, we ask LLMs to autoformalize a grade schoolmathematical word problem.
Remarkably, PaLM and Codex are both capable of formalizing thestatement perfectly.
This is surprising because formalizations of grade school math problems ininteractive theorem provers are rare (if they exist at all), as this type of mathematics is not of interestto formal mathematicians.
Even more, none of the examples in the prompt (see Appendix ?? ) that weprovide are of this type.
It is hence remarkable that the model is capable of extrapolating to this typeof statement, showing a great promise of using LLMs for autoformalization.
To study this problem in more depth, we probe PaLM models of various sizes (8B, 64B, 540B) withoutputs shown in Appendix ?? , and notice that scale is crucial for the LLMs ability to formalize.
We problem from IMO 1987.
Case Study 2 Question:
Natural Language version: " When all the girls at Madeline’s school line up in rows of eight, there are seven left over.
If instead they line up in rows of four, how many are left over?
The ﬁnal answer is 3.
" Translate the natural language version to an Isabelle version:
PaLM output: theorem fixes n::nat assumes "n mod 8 = 7" shows "n mod 4 = 3"
Case Study 3 Question:
Natural language version: "Let f be a linear function for which f (6) − f (2) = 12 .
What is f (12) − f (2)?
The ﬁnal answer is 30.
" Translate the natural language version to an Isabelle version:
Codex output: theorem fixes f :: "real \<Rightarrow> real" assumes "linear f" "f 6 - f 2 = 12" shows "f 12 - f 2 = 30" observe that the 8B and 64B models are incapable of formalizing this problem, but the largest 540Bmodel is able to produce a correct formalization.
Case Study 3 (Figure 2)
In our third case study, Codex gives an incorrect formalization in Isabelle.
The mathematical statement involves a concept of “linear function”, which the model fails to formalizecorrectly.
Codex assumes this is already a known concept in Isabelle, and made up a name: linear f .
Can the model learn to formalize such problems if the prompt contains an example that explains theconcept of a line?
We explore this and give an afﬁrmative answer to the question (see Appendix ?? ).
Once seeing a tangentially related problem that explains the concept of a “line”, Codex is able toperfectly formalize a “linear function”.
This shows the importance of the few shot examples weinclude, and also how good a few-shot learners these models are!
Has the model memorized these formalizations?
Whilst we do not have access to the trainingset of Codex, we attempted to ﬁnd any occurrences of the formalizations produced in the case studieson the internet.
We Googled them in different variants and inspected the ﬁrst page of the searchresults.
We tried variants with and without an “Isabelle” preﬁx, with and without quotation marksand other special characters, and also individual parts of it, such as “ Isabelle "n mod 8 = 7" ”,but we did not ﬁnd any occurrences of related statements.
We also tested that we are indeed ableto ﬁnd occurrences of Isabelle formalizations on the web with this methodology, using pieces offormalizations picked from several websites, including the Archive of Formal Proofs.
Hence, we areconﬁdent that the model has not memorized the formalizations it generated.
4.3 BLEU for Model Comparisons
The miniF2F benchmark contains 140 algebra problems and 120 number theory problems fromthe MATH dataset.
For these problems, we have human ground truth formalizations in Isabelle,which gives us an evaluation set with pairs of natural language statements (from MATH) and theirformalizations.
We use this dataset to quantitatively compare different LLMs.
Given the observation about few shot learning in Case study 3, we decided to add more relevantexamples to each subject to improve the quality of autoformalization.
For each subject (i.e., algebraand number_theory ), we randomly sample 10 problems to construct the few shot prompt.
The restof the problems are used for evaluation (i.e., 130 for algebra and 110 for number_theory .
Weprovide the prompt used in the Appendix ?? and ?? .
We use PaLM models of varying sizes and Codex to perform the autoformalization, and compute the
BLEU scores of the formalizations, shown in Table 1.
Conﬁrming our observation in Case study 2,we see a clear trend that scaling improves translation, as the BLEU scores consistently improve whenwe scale PaLM models from 8B to 540B, for both subjects.
In addition, we see that the Codex model
[Table] [Table]
is better at autoformalization measured by BLEU, possibly due to the fact that Codex was trained onmore formal data than PaLM.
4.4 Human Evaluation of Failure cases
To better understand LLMs’ ability to do autoformalization, we manually inspect Codex’s autoformalizations of 150 random problems from the MATH dataset [24].
50 of the problem statementsare sampled from the algebra training set, 50 from number_theory and 50 from intermediate_algebra .
For algebra and number_theory , we use their corresponding prompt as in the lastsection, shown in Appendix ?? and ?? .
For intermediate_algebra , we use the prompt we usedfor algebra (Appendix ?? ).
We classify the failure modes of these translations, shown in Table 2.
We see that out of 150 problems, Codex is capable of translating 38 problems perfectly – a successrate of 25.3 % .
The majority of the failures are due to the misalignment of informal and formaldeﬁnitions.
For example, when seeing the phrase “the greatest possible value”, the LLMs often failto align it with the function Greatest / Max in Isabelle.
Another example is the failure to align thefactorial of n (i.e., !
n ) to fact n in Isabelle.
Other common failure modes include the misapplicationof functions (e.g., applying a preﬁx function in an inﬁx way).
Autoformalization for Neural Theorem Proving
To demonstrate the usefulness of the formalized statements, we explore if one can improve neuraltheorem provers by training the neural models on proofs of automatically translated theorems.
In thissection, we combine autoformalization with expert iteration algorithms [4], and achieve a new stateof the art in miniF2F benchmark.
5.1 Expert Iteration with Autoformalization
The basic idea of expert iteration [4] is to iteratively generate a better dataset using the model, anduse the data to improve the model quality.
This allows the model to generate an even better quality ofthe dataset and hence a better model, forming a self-improvement cycle.
In neural theorem proving, one way to get better quality data is to use feedback from the proofchecker to run many proof searches (or generate multiple proofs) and check the proof attempts forcorrectness.
Newly found correct proofs can then be used as the new training data to improve theneural prover [7, 35, 36].
The main critical ingredient that is needed is a set of problem statements onwhich the model can perform proof search to obtain new training data.
However, unlike in Polu et al.
[Table]
[36], where one asks humans to manually formalize a set of problems to get formal statements, herewe use LLMs to autoformalize the theorems in order to kick off the self-improvement cycle.
More formally, denote a base neural theorem prover as M 0 .
Let the set of autoformalized problemsbe A .
For each iteration i = 1 . . .
N , we carry out the following procedure: use the language model
M i − 1 with best-ﬁrst search to prove as many theorems as possible in A , collect the set of successfulproofs S i , concatenate successful problems from all iterations with the formal mathematics problemsto create the set
A i =
( (cid:83) j ≤ i S i ) ∪B , and ﬁne-tune M 0 on it for exactly one epoch to get a new model
M i .
When we take the union of successful proofs from all past iterations, we perform deduplicationby problem statements, similar to Polu et al.
5.2 Neural Theorem Provers
To demonstrate the effectiveness of the approach, we start with a recently introduced neural theoremprover for Isabelle, LISA [26].
The LISA agent is ﬁne-tuned on the PISA dataset [26] (extractionand interaction code under a BSD license), which consists of 2.49 million proof steps from the
Isabelle/HOL library (under a BSD-style license) and the Archive of Formal Proofs (under variouslicenses as described here).
The model is trained with the objective to predict the next token in a proofstep, given the proof state and the last proof step.
Following the setup of Thor [3], which achieves
SOTA performance in the no-additional-training-data category on MiniF2F, we invoke Sledgehammerwith a 30 second timeout when the model generates a step containing any of the keywords metis,meson, and smt .
We use Wang [45]’s implementation (under an Apache license 2.0) of a GPT-2 [38] style decoder-onlytransformer [44] model with 700M non-embedding parameters.
The model has 24 layers, 24 attentionheads, a hidden dimension of 1536, and a vocabulary size of 50400.
We pre-train the model on the
GitHub + arXiv subsets of The Pile [14] for 200,000 steps, with a context length of 2048 tokens.
Inpre-training we use a warmup strategy [18], raising the learning rate linearly from 0 to 2 × 10 − 4 in3,000 steps.
We then use a cosine learning rate scheduler [34] for the rest of the pre-training, witha ﬁnal learning rate of 1 . 2 × 10 − 5 .
We use a global batch size of 32 sequences, or 65,536 tokens.
For ﬁne-tuning we use the same learning rate schedule, with 10,000 warmup steps, 90,000 annealingsteps, maximum learning rate 3 × 10 − 4 and ﬁnal learning rate 3 × 10 − 5 .
The global batch size is 144sequences, or 294,912 tokens.
The model’s evaluation loss reaches a minimum after 13,000 steps andwe use that checkpoint.
For the ﬁne-tuning in expert iteration, we ﬁx the learning rate at 3 × 10 − 4and the batch size at 144 sequences, and train the model for exactly one epoch.
Here we give the details regarding the best-ﬁrst search strategy used in evaluation: we maintain apriority queue of search nodes with queue length 32 .
The accumulated log probability of the proofsteps is used as the queue priority.
For each theorem to prove, we ﬁrst initialize the queue with onenode that has the theorem declared and no proof step applied to it.
At each time-step, we dequeand sample 32 possible proof steps to apply to the node. The nodes corresponding to steps thatsuccessfully proceed the proofs then get added to the queue. We repeat this process until the theoremis successfully proven, or we reach our computational budget. Machine speciﬁcation For pre-training, ﬁne-tuning, and evaluation, we use a TPUv3 with 8 coresfrom Google Cloud Platform.
The Isabelle process has access to up to 32 CPU cores.
We estimatethat running all the experiments in this paper requires a total of 780 TPU hours.
Question: An arbitrary product of compact spaces is compact.
Codex Output: theorem fixes X :: "’a set set" assumes "\<forall
>x \<in>X. compact x" shows "compact (\<Prod> x \<in> X. x)"
5.3 Result
We use Codex with greedy decoding to formalize 3908 mathematical problems in algebra , intermediate algebra , and number theory from the training set of MATH [24], with the same fewshot prompts used in Section 4.4.
Out of them, 3363 of the autoformalized theorems are syntacticallycorrect.
We then perform expert iteration on this dataset.
We start with a neural theorem prover ( M 0 ) as described in Section 5.2.
In our ﬁrst iteration, M 0proves 782 theorems, with a success rate of 23 .
3% (out of 3363).
This gives us a new set of veriﬁedproofs to further train the neural theorem prover.
We proceed to ﬁne-tune our neural theorem proverin the fashion described in Section 5.1 to get a new prover ( M 1 ).
This process is repeated in thesecond iteration, giving us 1011 successful proofs from the autoformalized theorems ( 30 . 1% ).
Weﬁne-tuned M 0 again on all available deduplicated proofs to obtain M 2 .
After each stage of ﬁne-tuning, we evaluate the neural theorem prover on miniF2F [52].
The resultsare shown in Table 3.
The base model ( M 0 ) has a success rate of 28 . 3% and 29 .
9% on the validationand test fractions of miniF2F respectively.
It can be observed that the ﬁrst expert iteration increasesthe success rate of the neural prover by 7 . 8% and 4 .
1% to 36 .
1% and 34 .
0% on the valid and testsets.
The second iteration further improves them both by 1 . 2% , to 37 .
3% and 35 .
By doing twoexpert iterations on the autoformalized theorems, the neural prover achieves a success rate that is5 .
6% higher than the previous state-of-the-art.
An Outlook on Autoformalizing Advanced Mathematics
So far, we focused on mathematical competition problems, in which we achieve signiﬁcant resultsusing autoformalization.
Not only can LLMs autoformalize non-trivial theorems, the autoformalizedtheorems can also improve neural prover performance.
In this section, we take a peek into moreadvanced mathematics.
We hope to identify some of the limitations of our methods when it comes toautoformalizing mathematics in the wild.
6.1 Autoformalization: from Natural Language to
Isabelle
Autoformalization is extremely challenging in the sense that the model needs to (1) bridge the logicalgaps left in pen-and-paper proofs, (2) assume the implicit contexts and assumptions, and (3) aligninformal deﬁnitions/concepts to formal ones.
This task is further complicated considering the contextcan change from time to time and the same mathematical object can be formalized in subtly differentways.
In this paper, we only deal with autoformalization of theorem statements, where the modelis mainly challenged by deﬁnition alignments.
Compared to competition problems in Section 4.4,advanced mathematical statements assume more context information that additionally obscures thealignment.
Figure 3 shows a typical case where the model fails to align the informal deﬁnition to the formalone.
Based on the notation used, the model does not correctly distinguish between products ofnumbers, products of sets, and products of topological spaces.
In Isabelle, the correct conclusionfor the statement should be compact_space (product_topology X I) , where I is an index set thatought to have been introduced in the fixes and assumes sections.
lemma seteqI:
[Equation]
= ⇒
[Equation]
Translate the Isabelle version to a natural language version:
Codex Output: "If A and B are sets such that A ⊆ B and B ⊆ A , then A = B ."
lemma topologyI: " [Equation] [Equation] [Equation]
⇒ is_ [Equation] _open
[Equation]
⇒ is_open T ( (cid:83) [Equation]
[Equation]
= ⇒ topology T"
Translate the Isabelle version to a natural language version:
Codex Output: "If T is a set and T is closed under ﬁnite intersections and arbitrary unions, then T is a topology."
6.2 Informalization: from Isabelle to Natural Language
So far, we explored one direction of translation: from natural language statement to formal statementin Isabelle.
The other direction, informalization , is also of great importance for two reasons: (1)the informal texts are much easier for humans to comprehend and communicate, and (2) we canalign translated informal statements with formal ones to create data, and use the back-translationtechniques [41] to potentially boost the translator’s performance further.
In this section, we explore
Codex’s capability of translating formal Isabelle statement to natural language. A corpus of 38 formal-language theorems, lemmas, and deﬁnitions is selected by an Isabelle expert. These statements are automatically translated to informal mathematics using Codex; to see theprompt we used and the results for all 38 examples, see Appendix ?? and ?? .
We present twoexamples of informalization in Figure 4.
Of the 38 examples, 36 were translated to a reasonablycoherent statement, and 29 of these statements (76%) were more-or-less correct, giving a vastly bettersuccess rate than the 25% success rate of formalization (Section 4.4).
Our main conclusion is thatfor advanced mathematics, the model is better at informalization than formalization, showing theprospect of backtranslation style algorithms.
Note that the standard is more relaxed here since we assume a human reader will supply the obviouscontext and correct mistakes when the intended meaning is obvious (intended by the hypotheticalhuman writer of these sentences).
To illustrate, an example of a minor “acceptable” error: assumingthat “ w, z are in the same connected component of the plane” when, in context, it is clear thatw, z should be assumed to be in the same connected component of the complement of a previouslyspeciﬁed curve.
(The assumption as originally stated is trivial.)
For an example of a major error:almost-perfect translation of the Central Limit Theorem that omits the assumption of identicaldistributions.
Discussion
Promise of Autoformalization with LLMs We have seen that automated formalization of informally given natural language statements is generally possible, even with language models nottrained for this particular task.
Also, automatically formalized statements are useful for training andimproving the reasoning capabilities of automated neural provers.
Our hope is that improved versionsof this methodology will be capable of enabling a positive feedback loop involving formalization andformal reasoning that has the potential of reaching human level capabilities in both respects, as wassuggested by [42].
Limitations and future directions
We use a static model for the formalization process.
For largescale autoformalization, we will need to formalize larger theories, preferably without ﬁne tuning themodel, as training it could be cumbersome and resource consuming.
However, in order to utilizethe newly added notions, the model would need to keep whole large theories in the current contextwindow, which exceeds those of the current LLMs.
This limits our approach to the generation offairly small pieces of formal mathematics and the automatic formalization of entire theories including their deﬁnitions will require new research ideas.
One path towards this goal might be the use ofcontinuous training or expert iteration, cycle-consistency-based training [30, 46], or novel uses ofin-context learning.
To generate larger theories we will also need neural networks that can recalllonger sequences (current LLMs are typically limited to a few thousand words).
Retrieval-augmentedlanguage models, such as the memorizing transformer [51] offer one path to overcome this limitation.
Societal Impact While the potential of creating negative societal impact through formalizations issmall, the use of LLMs always comes with risks.
For example, for deploying an autoformalizationtool using LLMs we would need to consider the inclusivity of variable and lemma names, and of theattribution of scientiﬁc ideas.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See discussion section.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
discussion section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
[Table] (c) Did you report error bars (e.g., with respect to the random seed after running experi- [Table]
ments multiple times)?
[No] It is expensive to run multiple times, and we believe
the
[Table]
results are signiﬁcant.
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5.2.
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited the creators
[Table]
of the assets
when we ﬁrst mentioned them.
[Table]
(b) Did you mention the license of the assets?
[Yes] We included the links to the licenses [Table]
of the assets
when we ﬁrst mentioned them in Section 5.
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]We include the code we used to train the models in the supplemental material.
The datawe used are all under open-source licenses.(d)
Did you discuss whether and how consent was obtained from people whose datayou’re using/curating?
[Yes] The assets we used mostly have open-sourced licenses asmentioned previously. [Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[No] The data we used are mathematical proofs sowe think it is apparent that they do not contain personally identiﬁable information orany offensive content.5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation?
[N/A]
[Table]