Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Provers
Anonymous Author(s)
AffiliationAddress email
Abstract
In theorem proving, the task of selecting useful premises from a large library tounlock the proof of a given conjecture is crucially important.
This presents achallenge for all theorem provers, especially the ones based on language models,due to their relative inability to reason over huge volumes of premises in textform.
This paper introduces Thor, a framework integrating language models andautomated theorem provers to overcome this difficulty.
In Thor, a class of methodscalled hammers that leverage the power of automated theorem provers are used forpremise selection, while all other tasks are designated to language models.
Thorincreases a language model’s success rate on the PISA dataset from 39% to 57% ,while solving 8 .
2% of problems neither language models nor automated theoremprovers are able to solve on their own.
Furthermore, with a significantly smallercomputational budget, Thor can achieve a success rate on the MiniF2F dataset thatis on par with the best existing methods.
Thor can be instantiated for the majorityof popular interactive theorem provers via a straightforward protocol we provide.
Introduction
In theorem proving premise selection is the task of identifying useful facts from a large librarythat enable finding the proof of a given conjecture.
It is essential for the discovery of many proofs,and Automated Reasoning in Large Theories (ARLT) depends on having apt methods for premiseselection
[Kühlwein et al., 2012, Sutcliffe et al., 2007].
A group of proof methods have been developedinside interactive theorem provers (ITPs) to deal with this task.
They use external automated theoremprovers (ATPs) to reach the remaining goals, inspect the proofs produced and pick out the premisesinvolved in them.
Such systems are called hammers [Blanchette et al., 2016].
Hammers are availablein many ITPs [Paulson, 2010, Kaliszyk and Urban, 2015, Gauthier and Kaliszyk, 2015, Czajka and
Kaliszyk, 2018] and are immensely popular within the theorem proving community.
Language models have had some successful applications in the area of formal theorem proving [ et al.
However, we observe that language-model-based reasoning systems are inept at premise selection.
The difficulty of premise selection for language models is that they cannot effectively reason over thousands of available facts and their definitions in plain text form.
In subsection 2.2, we elaborate on the scale of the problemslanguage models need to deal with for premise selection and provide empirical evidence for thisdifficulty.
Seeing that hammers are very good at finding relevant facts, we propose in our frameworkto offload the premise selection task to them from language models.
The resulting system is Thor, aframework that organically integrates language models and ATPs via the use of hammers.
The methodology of Thor is simple and can be deployed in any hammer-enabled ITP: we first use the hammer method to attempt to prove every proof state in the training problems, and mark the successful application steps.
Then we train the language model on the training problems, predicting a special token (e.g., <hammer> ) if the hammer can be applied at the state.
When doing evaluation, ifthe language model emits the special token, we invoke the hammer method.
This methodology incursvery little extra computation compared to standard language model training while capitalising on thepotential of a hybrid neuro-symbolic model.
To demonstrate the usefulness of Thor, we instantiate it with a language-model-based reasoningsystem for the ITP Isabelle and Sledgehammer [Paulson, 2010], Isabelle’s implementation of thehammer method.
We then investigate the performance of the instantiated Thor system on two datasets,
PISA [Jiang et al., 2021] and MiniF2F [Zheng et al., 2022].
On PISA we dramatically improve thesuccess rate of a language-model-based reasoning system from 39 .
0% to 57 .
0% and solve 8 .
2% ofproblems that cannot be solved by either language models or Sledgehammer alone.
On MiniF2F, Poluet al.
[2022] used expert iteration to improve on a language model and achieved the state-of-the-art1-pass success rate of 29 . 6% .
With much less computation, Thor increases this rate to 29 .
9% , slightlyexceeding the previous result.
It is worth noting that Thor and expert iteration can be used in tandem.
In this paper, we demonstrate that finding suitable sub-systems for premise selection can benefitthe performance of the overall reasoning system.
Given Thor’s strong performance, computationalefficiency, and applicability to many ITPs, we believe it should become a strong baseline as often aspossible when language models are used for theorem proving.
Contributions
We created Thor, a theorem proving framework which integrates language models and
automated theorem provers via using hammers.
We raised the state-of-the-art success rate of language-model-based reasoning systems on
PISA from 39 .
0% to 57 .
Thor proved 8 .
2% theorems which cannot be proved by eitherlanguage models or Sledgehammer.
We improved the state-of-the-art success rate on MiniF2F from 29 .
6% to 29 .
9% , matchingthe language models trained with expert iteration, but with far less computation.
Background2.1 Automated and Interactive Theorem Proving
Mechanising theorem proving has been a grand challenge of artificial intelligence since the late1950s [Gelernter, 1959].
A group of systems which we call automated theorem provers attempt touse automated procedures to determine the validity of conjectures (e.g., the DPLL algorithm
[Daviset al., 1962] for SAT problems [Tarski, 1969]).
Popular examples of ATPs include E, SPASS, Z3,
CVC4, and Vampire.
Although SAT is known to be NP-complete [Cook, 1971], modern ATPs canoften solve problems with millions of symbols [Ohrimenko et al., 2009] and are useful practically.
ATPs are often based on fragments of first-order logic, which limits the type of theorems they canexpress.
Hence, projects such as the formalisation of complicated mathematical results [Gonthieret al., 2008, Avigad et al., 2007, Gonthier et al., 2013, Scholze, 2021] and operating system kernelverification [Klein et al., 2009] are done in interactive theorem provers, often based on higher-orderlogic or dependent type theory.
ITPs and ATPs have very different objectives: ITPs aim at making iteasy to formalise a diverse set of problems in numerous mathematical domains in a sound manner;while ATPs focus on improving the efficiency and performance on very well-defined problems like
SAT solving.
Prominent ITPs include Isabelle, Mizar, HOL Light, HOL4, Lean, and Coq.
Theoremproving in ITPs can be modelled as a sequential decision process: initially a theorem gets declaredand the proof state contains some goals; at each step, the user produces a proof step thatapplies to and transforms the proof state ; when all the goals have been discharged, the theorem isconsidered proven.
Large libraries of mathematical knowledge such as the Archive of Formal Proofs 1and the Mizar Mathematical Library 2 have been built around these ITPs.
Because of the size of thesemathematical libraries, to find useful premises in them is a difficult problem.
In the next subsectionswe illustrate how two different approaches deal with premise selection.
[Figure]
Models
Proving
Language models that automate theorem proving mostly follow the approach of the GPT-fmodel [Polu and Sutskever, 2020]: pre-trained causal language models are used to predict a proofstep that can be applied, given the current proof state and some optional context .
Concretely,a language model can take as input and output, two sequences of the following form:
INPUT
[Equation] [Equation]
At test time, the reasoning system receives the text representation of the current proof state ,samples a proof step from the language model, applies it to the ITP, and repeats until the proofis finished or a computational budget has been reached.
A best-first strategy is often used for proofsearch: a queue of search nodes is maintained with the priority being the accumulated log likelihoodof the generated proof steps .
Language models treat all input and output information as text and they are usually limited to be a fewthousands of characters long.
To do premise selection well, the language model has to either memoriseall the relevant premises during training, or be prompted with available premises in evaluation.
Itis difficult to do the former because a mathematical corpus can have too many facts for a languagemodel to remember.
For example, the Archive of Formal Proofs has more than 200,000 theorems,plus the numerous definitions and their derivations to serve as premises.
The latter is no easierbecause there may be too many premises to fit into the input.
For instance, if we use the textualrepresentation of 300 available premises (a usual number used for premise selection with symbolictools) and their definitions as the context in the input-output format above, the input length canwell exceed 10,000 characters and the limit of standard language models.
We observe that empirically1 .
9% of the steps involving premise selection generated by the language model manage to advancethe proof, while the number is 28 . 2% for steps having no premises.
Hence, a good mechanism forpremise selection could bring crucial benefits.
2.3 Hammers
Blanchette et al.
[2016] define hammers as methods that “automate reasoning over large librariesdeveloped with formal proof assistants (ITPs)”.
Consider, for example, Sledgehammer (designedfor Isabelle) which is the original and the most popular implementation of hammers.
Figure 1presents a proof of √ 2 / ∈ Q in Isabelle.
The beauty of using Sledgehammer with Isabelle is thatdespite the complicated-looking proof, humans only need to sketch the proof in Figure 1a and let
Sledgehammer find all the necessary premises to complete every single proof step.
The final acceptedproof is presented in Figure 1b.
The Sledgehammer proof steps use the internal proof methods metis, meson, smt, auto, simp, fastforce and blast .
Conveniently, this tells us whichsteps in the corpus are generated by Sledgehammer.
Note that a human user might also use the proofmethods auto, simp, fastforce and blast as these do not contain additional premises.
Onlythe methods metis, meson, smt are exclusive to Sledgehammer.
We now describe how Sledgehammer performs premise selection: Sledgehammer makes it possibleto leverage the advancement of ATP research while using ITPs, and can thus be seen as a bridgebetween the two [Paulson, 2010].
When invoked, Sledgehammer translates the current goal togetherwith hundreds of possibly relevant premises into a format (e.g., SMT-LIB, TPTP) that external
ATPs can understand [Meng and Paulson, 2008].
The ATPs are then executed to solve the currentgoal.
Note that Isabelle follows a kernel philosophy (i.e., only a handful of axioms and inferencerules are trusted), and external ATPs are used skeptically—should a proof be found by the ATPs,
Sledgehammer picks out the useful premises, and reconstructs the proof within the Isabelle kernel(e.g.
, using the primitive inference rules).
Here, external ATPs serve as relevance filters of premisesrather than trusted oracles.
Hammers implemented in other ITPs are largely similar.
Thor
In this section we introduce Thor, a framework integrating language models and automated theoremprovers via the use of hammers.
Thor is motivated by the difficulty for language models to do premiseselection and the excellent performance of hammers for it: we should be able to drastically improveautomation in theorem proving if we can take the best from both worlds.
Below we provide the protocol of adopting Thor for a hammer-enabled ITP.
We first provide Thor’straining data preprocessing procedure in Algorithm 1, and then look at a concrete example todemonstrate its use.
Algorithm 1 Thor’s training data preprocessing algorithm.
Require:
Proof state s , hammer method h
INPUT = s.input if h(s)
→ success then ▷ Hammer can be applied to the proof state
OUTPUT = <hammer> <EOS> else ▷ Hammer fails at the proof state
OUTPUT = s.output end if return (INPUT, OUTPUT)
Now consider the situation in the proof of √ 2 / ∈ Q (Figure 1) after the step then have "even a" :without Thor, it should produce the following datapoint
INPUT: <SOS> <CTXT> $(context) <
PRF_STT> $(proof state) <
PRF_STP>
OUTPUT: by (smt (z3)
even_power oddE) <EOS>
With Thor’s preprocessing, we apply the hammer method to the proof state and find out that it can bedone successfully.
Hence, we keep the input the same and change the output to:
OUTPUT: <hammer> <EOS>
If the hammer method cannot be applied, we leave the datapoint unchanged.
We iterate over everydatapoint in the training data and apply this preprocessing algorithm.
We hypothesise that being exposed to training data in this format, the language model is capable oflearning a heuristic for when the hammer can be successfully invoked.
At evaluation time, wheneverthe language model outputs the sequence <hammer> <EOS> , instead of applying it directly to the
ITP, we call the hammer method.
This effectively makes the hammer an invokable method for thelanguage model.
This protocol is straightforward to implement for hammer-enabled ITPs.
The only extra cost of deploying Thor is in the data preprocessing step.
Multiplying the hammertime limit by the average number of problems submitted to the Archive of Formal Proofs in one year,we estimate that 7400 CPU hours per year are needed to preprocess one of the largest proof corpora available.
This is a modest cost since the process only needs to be done once per dataset and the resultscan be shared.
Better still, for some ITPs, the hammer method leaves a trace, greatly reducing the timeneeded to figure out which steps can be solved by hammers.
For the ITP Coq, all steps containingthe keyword sauto are generated by CoqHammer [Czajka and Kaliszyk, 2018].
For Isabelle, allsteps containing the keywords metis, meson, smt are generated by Sledgehammer (describedin Section 2.3).
With these traces, deploying Thor on ITPs like Coq or Isabelle incurs little extracomputational cost compared to training a standard language model.
Experiment
Our experiments are intended to answer the following research questions:
Can Thor prove theorems that cannot be proved by language models or automated theorem
provers individually?
Does Thor improve premise selection for language models?
Does explicitly learning how to select premises hurt the performance of language models?
How important are the context information and the diversity of sequence generation?
How does Thor compare with other methods at improving language models for theorem
proving?
To answer these questions, we create an instance of Thor for the ITP Isabelle.
We choose Isabelle fortwo reasons: (1) Isabelle’s Sledgehammer is one of the most mature hammer methods among major
ITPs, and may thus showcase Thor’s full potential; and (2) Isabelle’s Archive of Formal Proofs is oneof the world’s largest formal mathematical libraries, suitable for data-hungry methods like languagemodels.
We make explicit the details of our experimental setup next.
4.1 Experimental Setup
Machine specification For pre-training, fine-tuning, and evaluation, we use a TPUVM with 8 coresfrom Google Cloud Platform.
The Isabelle process has access to up to 32 CPU cores.
We estimatethat reproducing all the experiments in this paper requires a total of 1160 TPU hours.
Language model architecture We use a decoder-only transformer [Vaswani et al., 2017] languagemodel, adapting the setup, codebase, and hyperparameters from [Wang and Komatsuzaki, 2021].
Thelanguage model has 700 M non-embedding parameters, with 24 layers, 24 attention heads, a hiddendimension of 1536 , and a GPT-2 [Radford et al., 2019] tokenizer with a vocabulary size of 50400 .
Rotary positional embeddings [Su et al., 2021] are used.
The model is pre-trained on the GitHub +arXiv subsets of The Pile [Gao et al., 2021], with a context length of 2048.
We use a global batch sizeof 32 sequences which amounts to 65536 tokens.
For the first 3,000 steps, the learning rate linearlyincreases from 0 to 0 .
0002 , and then it follows a cosine schedule with a final value of 1 . 2 × 10 − 5for 197,000 steps.
We use a weight decay rate of 0 .
05 and no dropout for pre-training.
Pre-trainingtakes ≈ 150 TPU hours.
For fine-tuning, we use the procedure described in Section 3 to prepare the
PISA dataset.
We use the most recent proof step as the context in each datapoint.
The samelearning rate scheduling strategy is used, with a peak learning rate of 3 × 10 − 4 after 10,000 steps anda final learning rate of 3 × 10 − 5 after a further 90,000 steps.
We use a dropout rate of 0 .
15 and aweight decay rate of 0 . 1 .
The global batch size is 256 sequences, or 524 , 288 tokens.
We early-stopfine-tuning and take the checkpoint at 11,000 steps for evaluation as the validation loss reaches aminimum then.
Fine-tuning takes ≈ 50 TPU hours.
Sledgehammer configuration To set up Sledgehammer, we mostly follow the default Isabelle2021configuration.
An important default parameter is that the Sledgehammer timeout limit is 30 s. Ourconfiguration uses the on-machine versions of the five default ATPs (E, SPASS, Vampire, Z3, and
CVC4) to prevent performance deviation caused by network issues.
Proof search To sample from the language model, we use temperature sampling with the temperature parameter T = 1 . 2 .
To search for the proof of a theorem, we use the best-first search strategydescribed in [Polu and Sutskever, 2020].
The queue is ordered by the accumulated log likelihoods ofthe generated proof steps , with a maximum length of 32.
Each proof step has a timeout limit
[Table]
of 10 s.
The search is terminated if and only if one of the following scenarios happens: (1) a validproof has been found for the theorem; (2) the language model is queried 300 times; (3) a wallclocktimeout of 500 s has been reached; (4) the queue is empty but the theorem is not proved.
Empirically,it takes ≈ 60 TPU hours to evaluate 1 , 000 problems.
Our language model setup is different from Language models of ISAbelle proofs [Jiang et al., 2021,
LISA] in three aspects: (1) our language model has 700M instead of 163M non-embedding parameters(2) the most recent proof step is included in the language model prompt (3) a higher samplingtemperature ( 1 . 2 instead of 1 . 0 ) is used.
4.2 Datasets and Environment
We use two datasets.
The first is the PISA dataset [Jiang et al., 2021], which includes the Isabelle/HOLrepository 3 under a BSD-style license and the Archive of Formal Proofs version 2021-10-22 4 , whosevarious entries are under open-source licenses as described on its official page.
PISA contains the corehigher-order logic library of Isabelle, as well as a diverse library of proofs formalised with Isabelle,mostly concerning mathematics or verification of software and hardware.
The PISA dataset contains2 .
49 million datapoints in total.
The proof states have an average length of 369 characters andthe proof steps have an average length of 33 characters.
All of the Isabelle/HOL theorems gointo the training set as they are considered foundational and might be used by all other repositories.
We make a 95% / 1% / 4% split of theorems from the AFP for the training/validation/test sets.
Werandomly select 3,000 theorems from the test set ( PISA/test ) for the evaluation of model performance.
The second is the Isabelle fraction of the MiniF2F dataset [Zheng et al., 2022] under an Apache license.
The dataset contains 488 high school mathematics competition problems split into a validation setand a test set, each with 244 problems.
These problems have been formalised in Lean, Metamath,and Isabelle to provide a benchmark of the same problems in different ITP languages.
This allows usto contrast different approaches developed for different ITPs.
Since we do not use the validation setfor model selection, we do not actually distinguish between the two sets.
Hence, we mainly comparewith previous work on the test set as the final result.
We use the codebase by Jiang et al.
[2021], under a BSD 3-clause license, to interact with the Isabelleserver and prove theorems from both datasets.
4.3 Thor Against an Ensemble of a Language Model and Sledgehammer
Because Thor has both a language model and Sledgehammer at its disposal, we wish to investigatehow it fares against a simple ensemble of the two.
We set out to evaluate the performance of Thor,as well as a language model of the same configuration, and Sledgehammer with a 120s timeout on
PISA/test .
It takes ≈ 50 TPU hours to evaluate Thor for 1000 problems.
The proof success rateson PISA/test are presented in the second column of Table 1.
We can see that the language modelalone and Sledgehammer alone can prove 39 .
0% and 25 .
7% of the problems respectively.
When wetake the union of the problems they manage to solve individually, we get a 48 .
8% success rate.
Thormanages to prove 57 .
0% of the problems.
This implies that for 8 .
2% of the problems, Thor uses boththe language model and Sledgehammer to complete the proofs, and it’s not possible to achieve this
[Figure]
[Table]
with only the language model or only Sledgehammer.
We perform 4 case studies on problems thatonly Thor can solve in Appendix A.
Thor’s motivation is to solve the premise selection problem for language models.
To confirm that Thorhelps premise selection, we collect the proofs generated by the language model and Thor respectivelyand count the number of premises in them.
The results are presented in Figure 2a: we can see thatfor proofs requiring 0 or 1 premises, Thor and the language model perform similarly.
But for proofsrequiring more premises, Thor performs much more robustly, finding several times more proofs thanthe language model.
We also count the number of premises in the ground truth proofs (written byhumans) for theorems the language model and Thor can prove.
The results are presented in Figure 2b:we see that whatever the number of premises the ground truth uses, Thor outperforms the languagemodel in finding proofs, and the more premises the ground truth proof has, the more obvious is theeffect.
We conclude that Thor is indeed more capable of premise selection than language models.
4.4 The Effect of Learning How to Select Premises
The procedure we described in Section 3 ensures that the language model learns when to do premiseselection, but not how to do it, by replacing the premise selection steps with <hammer> .
Here weinvestigate the effect of making the language model learn both when and how .
An easy way to achievethis is to create a variant of Thor: (i) at training time, use the original data; (ii) at evaluation time,when the language model outputs a sequence containing any of the Sledgehammer keywords, invoke
Sledgehammer.
This further simplifies data preparation and explicitly subjects the language model toperform premise selection.
To investigate the effect of this alternative approach, we evaluate a systemtrained in this way on PISA/test and present its success rate in Table 2.
We can see that it achieves asuccess rate of 55 .
4% on PISA/test , 1 . 6% lower than the base version of Thor, which suggests thatexplicitly learning how to do premise selection marginally decreases its success rate.
This result isexpected: since finding how to do premise selection is entrusted to the hammer method, the languagemodel should focus on learning when to invoke the hammer for optimal performance.
Making thelanguage model learn an irrelevant additional task only hurts Thor’s performance.
4.5 The Effect of the Proof Context
Our language model setup differs from that of LISA [Jiang et al., 2021] in that we use the mostrecent proof step as the context in the input data, as introduced in Section 3.
This is based onthe intuition that the most recent proof step information is beneficial for the language model’s
[Table]
reasoning ability.
In this subsection we perform an ablation study to confirm the effect of thiscontext on Thor.
Here a variant of Thor is trained without the context information and evaluatedon PISA/test .
The results are in Table 2.
We observe that this variant manages to prove 53 . 6% oftheorems on PISA/test , 3 . 4% fewer than the base version of Thor.
The drop in success rate indicatesthat the context information we use is crucial for the optimal performance of Thor.
4.6 The Effect of the Sequence Sampling Diversity
Our language model setup differs from LISA [Jiang et al., 2021] also in the sampling temperature.
Previous works on language models for theorem proving often use a temperature T = 1 .
[Polu and
Sutskever, 2020, Jiang et al., 2021] for sampling output sequences, while we use T = 1 . 2 .
A highertemperature in the sampling procedure means that the generated sequences are more diverse (havinga higher entropy).
Here we perform an ablation study on the diversity of Thor-generated sequences.
We evaluate Thor with sampling temperature T = 1 . 0 on PISA/test and the success rate is in Table 2.
We can see that the success rate with sampling temperature T = 1 . 0 is 55 .
7% , 1 . 3% lower than with
T = 1 . 2 .
This suggests a more diverse sampling strategy can improve Thor’s performance, and thatthe optimal diversity in language model samples varies for different systems.
4.7 Comparing Thor with Expert Iteration
There exist other methods for improving language models for theorem proving like value functiontraining [Polu and Sutskever, 2020], proof artifact co-training [Han et al., 2021, PACT], and expertiteration [Polu et al., 2022].
We wish to compare Thor with them.
However, these methods operate in
ITPs other than Isabelle and are thus hard to compare with directly.
Thankfully, Polu et al.
[2022]used expert iteration [Silver et al., 2017] to improve PACT [Han et al., 2021] and to achieve thestate-of-the-art result on MiniF2F, a dataset containing multiple ITP formalisations of the sameproblems.
Hence, we can fairly contrast expert iteration with Thor.
Expert iteration, as arguedby Polu et al.
[2022], provides an unsupervised curriculum for automated theorem provers: givensuitable data, it provides samples similar to the test distribution for further training, thus narrowingthe generalisation gap between training and test.
We should emphasise that Thor and expert iterationare not incompatible methods: one can use Thor together with expert iteration.
We start by evaluating Thor, a language model with the same configuration, and Sledgehammer on
MiniF2F.
The results are presented in Table 3.
We also include the success rates of the languagemodel that Polu et al.
[2022] used (PACT), as well as the language model after expert iteration in thesame table.
The success rates on the validation set are also included, but we use the rates on the testset as the final results, as the valid set can be used for model selection.
We can see that the languagemodel is able to prove 24 .
2% of the problems on MiniF2F, similar to PACT’s 24 .
Thor increasesthe success rate of the language model by 5 . 7% to 29 .
9% , while expert iteration increases the successrate of PACT by 5 .
% to 29 .
Hence, the improvement in proof success rate brought upon thelanguage model by Thor is comparable to that by expert iteration.
An important factor in choosing a suitable method is its cost.
Expert iteration requires manuallycreating a set of “curriculum” problems, evaluating the language model on them, and training thelanguage model on a growing training set for one epoch every iteration.
We estimate that to performexpert iteration at the same scale as Polu et al.
[2022] for Isabelle, it would cost 100 human hoursto formalise 300 maths problems, and 500 TPU hours to evaluate and fine-tune the language modelfor 8 expert iterations.
Thor, on the other hand, incurs little extra computational cost compared with training a standard language model.
We conclude that while requiring a much smaller computationalbudget, Thor can improve language models’ success rates to a similar degree as expert iteration.
Related Work
Language models were first applied to automate theorem proving by Polu and Sutskever [2020].
Since then, there have been a few works [Han et al., 2021, Jiang et al., 2021, Polu et al.
, 2022]aiming to enhance the ability of language-model-based reasoning systems, or to enable these systemsfor interactive theorem provers that were not supported before.
All of these works used the sameframework laid down by Polu and Sutskever [2020], namely to iteratively sample from a languagemodel and directly apply the output to the ITP.
Thor, to the best of our knowledge, is the first systemto explicitly hybridise language models and symbolic reasoning tools (ATPs) for theorem proving.
Instead of relying on language models entirely, Thor uses hammers, a well-established tool, to solvepremise selection.
With the growing bodies of formal mathematical libraries, premise selection has become one of themost crucial tasks of theorem proving.
The hammer method is one of the many ways that premiseselection can be done.
We have described how the Isabelle implementation of the hammer methodselects premises in Section 2. HOL(y)Hammer
[Kaliszyk and Urban, 2015] and CoqHammer [Czajkaand Kaliszyk, 2018] implement the hammer method for HOL Light and Coq respectively, making itpossible for Thor to be instantiated for them.
Apart from hammers, SInE [Hoder and Voronkov, 2011]and SRASS [Sutcliffe and Puzis, 2007] are both symbolic methods that take on the task of premiseselection by ranking the available premises according to their relevance to the current conjecture,measured by syntactic and semantic distances respectively.
MaLARea [Urban, 2007] pioneeredhaving machine learning components in premise selection systems and its later version MaLARea
SG1 [Urban et al., 2008] combines machine learning and formal semantics for premise selection.
Afew approaches [Irving et al., 2016, Wang et al., 2017, Kaliszyk et al., 2017] use deep learning in thepremise selection task.
These diverse methods may have merits over the hammer approach, and thushave the potential to be integrated as the premise selection component for future versions of Thor.
Discussion
In this paper we introduced a simple approach to overcome language models’ weakness in premiseselection for theorem proving: we created Thor, a framework that integrates language models andautomated theorem provers via the hammer proof method.
We presented a straightforward protocolfor deploying Thor on any hammer-enabled ITP (including Isabelle, HOL Light, Coq, etc.).
Theinstance of Thor with Isabelle dramatically increased the number of automatically proved theorems,suggesting that language models’ deficiency at premise selection can be effectively compensatedby utilising ATPs.
Furthermore, approaches like expert iteration [Polu et al., 2022] or proof artifactco-training [Han et al., 2021] have no contradictions and can be easily incorporated with Thor.
Compared with these methods, Thor has the additional advantage of being computationally efficient.
One limitation of Thor is that it only admits automated theorem provers that directly generate validproof steps in the ITP via the use of the hammer.
In Section 5, we pointed out that there are otherpremise selection tools with approaches different from the hammer method that the current version of
Thor cannot use.
Also, there exist methods which assist premise selection but do not directly generatethe proof steps.
An example of this is SErAPIS [Stathopoulos et al., 2020], which performs semanticsearch over the Isabelle mathematical library with the help of Wikipedia.
Thor cannot use this class ofmethods either.
We leave to future work the task of broadening the options for the premise selectiontool that Thor uses.
Here we only tested Thor on the ITP Isabelle due to the computational costs ofexperiments.
Therefore another future direction is to instantiate Thor with other ITPs and see whetherimprovements brought by Thor are as significant for other ITPs as we show them here for Isabelle.
Thor demonstrates how a difficult problem for language models can be solved by borrowing tools fromanother research domain.
We are encouraged by its success and think more problems like premiseselection can be solved similarly. E.g., Collins et al.
[2022] demonstrated how a symbolic plannercan make a language model more robust.
With its strong performance, computational efficiency, andconvenient deployment, Thor gives scope to tool hybridisation, which shows promise to be impactfulin the field of automated reasoning, and artificial intelligence in general.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We have clearly listed the contributions of the paper atthe end of Section 1.
These claims are justified in Sections 3 and 4.
(b) Did you describe the limitations of your work?
[Yes] We described the limitations
of our work in the second paragraph of Section
The limitations on the form
of thepremise selection tool and the uncertainty of full generalisation to other ITPs due
tocomputational budget constraint were discussed
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] As this
is a paper focusing on formal theorem proving, every mathematical result has been
checked by an interactive theorem prover and is unlikely to mislead the theorem proving community
Hence, we see no direct negative societal impacts of this work.
But we did
discuss the cost of training and evaluating language models in Section 4.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] I have.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] We have
included the urls to the implementations of language models we used, as well as the
code for generating data and interacting with Isabelle.
(b) Did you specify all the training details (e
g., data splits, hyperparameters
, how theywere chosen)?
[Yes] Yes.
We specified the training details in subsection 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running ex-
periments multiple times)?
[No] We did not include error bars because running the
experiments is very computationally expensive.
To evaluate a model for the full set of
test problems requires ≈ 180 TPU hours, empirically.
Attaining error bars by repeating
the experiments is prohibitively expensive for us.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We included the details of our
hardware setup, as well as the total estimated amount of compute in subsection 4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited the creators
of the assets
when we first mentioned them, extensively in subsections 4.1 and 4.2.
(b) Did you mention the license of the assets?
[Yes] We mentioned the licenses of the
assets we used in the paper when we first mentioned them.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We include the code we used to evaluate the models in the supplemental material.
The
data we used are all under open-source licenses.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] The assets we used mostly have open-sourced licenses.
We
mentioned that we had consent to adapt the proof from Li et al.
[2021] in Section 2.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] The data we used are mathematical proofs so
we think it is apparent that they do not contain personally identifiable information or
any offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not use crowdsourcing or conduct research with human
subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
We did not use crowdsourcing or conduct
research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
We did not use crowdsourcing or conduct
research with human subjects.
A Appendix
In this section, we present some lemmas solved by Thor only.
Case 1.
The lemma cols_upt_k _insert is from the QR Decomposition entry 5 in the AFP.lemma cols_upt_k_insert: fixes A::"’a^’n::{mod_type}^’m::{mod_type}" assumes k: "(Suc k)<ncols A" shows "cols_upt_k A (Suc k)
=
(insert (column (from_nat (Suc k))
A)
(cols_upt_k A k))" unfolding cols_upt_k_def apply (auto) apply (metis Suc_lessD from_nat_mono’ from_nat_to_nat_id
k less_Suc_eq_le
less_le ncols_def to_nat_le) by (metis from_nat_mono’ k less_imp_triv less_or_eq_imp_le ncols_def not_less_eq order_trans)
Here, cols_upt_k A (Suc k) returns the set of columns in the matrix
A up to the natural numberk+1 , while ncols A counts the number of columns in the matrix A .
In short, this lemma claims thatthe set of columns (in a matrix A ) up to column index k + 1 is equivalent to that of the same matrixup to column index k inserted with the ( k + 1) th column (of A ).
This will subject to the conditionthat k + 1 is less than the number of columns in A .
With Thor, the LM decided to unfold the goalwith the definition of cols_upt_k , which is followed by an auto tactic to simplify the proof state.
Allremaining subgoals are then discharged by Sledgehammer.
Case 2.
The lemma size_del_max is from the Weight-Balanced Trees entry 6 in the AFP.lemma size_del_max: "t ̸ = Leaf = ⇒ size t = Suc(size(snd(del_max t)))" apply ( induction [Table] del
[Table]
max.
induct
) apply simp apply (
clarsimp
split: prod.splits) apply
[Table]
smt z3) size
[Table]
rotateR size
[Table]
wbt.
[Table]
(1)) by simp
In this lemma, t is a weight-balanced tree, and the size function measures its size (as the namesuggests) and del_max deletes the maximum node from it.
Essentially, this lemma claims that when aweight-balanced its size will be reduced by one if we remove the largest node from it.
For the proof,
Thor intelligently performs structural induction with the induction rule del_max.induct and thensimplifies the proof state a few times, which includes splitting products with the rule prod.splits .
Finally, Thor concludes the remaining goals with Sledgehammer.
Case 3.
The lemma t_list_of_B_log_bound is from the AFP entry named as Priority Queues Basedon Braun Trees .
7lemma t_list_of_B_log_bound: "braun t = ⇒ t_list_of_B t ≤ 3 * (nlog2 (size t + 1) + 1) * size t" apply (induction t rule: measure_induct_rule[ where f=size]) apply (case_tac x) apply simp using braun.simps(1) t_list_of_B_braun_simps(1)
apply blast by (metis acomplete_if_braun
height_acomplete order_refl size1_size t_list_of_B_induct)
Here, size measures the size of a Braun tree; nlog2 stands for the function λx.
⌈ log 2 ( x ) ⌉ ;t_list_of_B is another measure of a Braun tree.
Basically, this lemma describes the relationship between a normal tree size and a Braun-tree specific measure.
The proof starts with an intelligentstructural induction, progresses with case analysis, and is concluded with Sledgehammer on each ofthe remaining subgoals.
5 QR_Decomposition/Gram_Schmidt.thy
[Table]
Case 4.
The lemma inj_imp_Ker0 is from the AFP entry named as Matrices, Jordan Normal Forms,and Spectral Radius Theory .
8lemma inj_imp_Ker0: assumes "inj_on
T (carrier V)" shows "carrier (V.vs kerT)
= { 0 V}" apply (rule equalityI) apply (rule subsetI) apply (unfold ker_def, auto) by (metis V.module.
M.zero_closed assms f0_is_0 inj_on_contraD)
Here, T is a linear map between two vector spaces.
The lemma claims that if the T is injective on thecarrier set of the space V , the kernel of T has to be a singleton set with the zero in V .
In this proof,
Thor naturally performs a sequence of introduction steps by applying the lemma equalityI andsubsetI , before unfolds the definition of a kernel (i.e., ker_def ) and uses auto to simplify the proofstate.
The final remaining goal is closed with Sledgehammer.
B Appendix
B.1 Thor calls sledgehammer cautiously and effectively
We gather some statistics to demonstrate the necessity of letting the language model learn whensledgehammer shall be applied.
In the training data, sledgehammer appears as the output for 2 .
40%of the steps.
During evaluation, sledgehammer execution accounts for 1 . 29% of the proof steps.
Ofthe 1 . 29% sledgehammer calls, 28 .
3% are successful.
B.2 Baseline experiment: calling sledgehammer for every new proof state
We run an additional experiment with the same setup as Subsection 4.3, but instead of letting thelanguage model decide when sledgehammer shall be called, we execute sledgehammer for every newproof state.
This resulted in a success rate of 51 .
7% on the same test suite, 5 . 3% lower than Thor.
Here we briefly analyse the reason for this inferior performance: Sledgehammer has a timeout of30 s, while an average step execution in Isabelle takes 7 ms.
The execution of 32 steps in a batch takes210 ms , 136 x faster than one sledgehammer execution.
Therefore, given a limited amount of time,the frequent querying of sledgehammer severely limits the proof search nodes the model can explore.
Hence it is detrimental to the model performance.