Original: L EARNING TO R EASON IN L ARGE T HEORIES WITHOUT I MITATION
Revised: L EARNING TO R EASON IN L ARGE T HEORIES WITHOUT I MITATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we demonstrate how to do automated higher-order logic theorem proving in the presence of a large knowledge base of potential premises without learning from human proofs.
Revised: In this paper, we demonstrate how to do automated higher-order logic theorem proving in the presence of a large knowledge base of potential premises without learning from human proofs.

Original: We augment the exploration of premises based on a simple tf-idf (term frequency-inverse document frequency) based lookup in a deep reinforcement learning scenario.
Revised: We augment the exploration of premises based on a simple tf-idf (term frequency-inverse document frequency) based lookup in a deep reinforcement learning scenario.

Original: Our experiments show that our theorem prover trained with this exploration mechanism but no human proofs, dubbed DeepHOL
Revised: Our experiments show that our theorem prover trained with this exploration mechanism but no human proofs, dubbed DeepHOL

Original: Zero, outperforms provers that are trained only on human proofs.
Revised: Zero, outperforms provers that are trained only on human proofs.

Original: It approaches the performance of a prover trained by a combination of imitation and reinforcement learning.
Revised: It approaches the performance of a prover trained by a combination of imitation and reinforcement learning.

Original: We perform multiple experiments to understand the importance of the underlying assumptions that make our exploration approach work, thus explaining our design choices.
Revised: We perform multiple experiments to understand the importance of the underlying assumptions that make our exploration approach work, thus explaining our design choices.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Theorem proving is a challenging benchmark for automated reasoning, and is an important milestone on the road to demonstrating that machine learning can produce a deep understanding of abstract concepts.
Revised: Theorem proving is a challenging benchmark for automated reasoning, and is an important milestone on the road to demonstrating that machine learning can produce a deep understanding of abstract concepts.

Original: In the long run, automated mathematical reasoning may become an important tool in engineering and scientiﬁc discovery.
Revised: In the long run, automated mathematical reasoning may become an important tool in engineering and scientiﬁc discovery.

Original: Due to their success in many other areas, neural networks have recently been considered as a way to guide theorem proving (Alemi et al., 2016;
Revised: Due to their success in many other areas, neural networks have recently been considered as a way to guide theorem proving (Alemi et al., 2016;

Original: Loos et al., 2017; Huang et al., 2019; Bansal et al., 2019; Paliwal et al., 2020) and demonstrate approximate mathematical reasoning abilities in latent space (Lee et al., 2020).
Revised: Loos et al., 2017; Huang et al., 2019; Bansal et al., 2019; Paliwal et al., 2020) and demonstrate approximate mathematical reasoning abilities in latent space (Lee et al., 2020).

Original: While there is only a relatively small number of fundamental proof rules (or proof tactics) applicable at any point in a proof, there is a very large number of premises (i.e., previously proven theorems and lemmas) that could be invoked.
Revised: While there is only a relatively small number of fundamental proof rules (or proof tactics) applicable at any point in a proof, there is a very large number of premises (i.e., previously proven theorems and lemmas) that could be invoked.

Original: The largest formalized libraries have over tens of thousands of theorems that can be used as premises.
Revised: The largest formalized libraries have over tens of thousands of theorems that can be used as premises.

Original: Thus, the main problem of reasoning in large theories is to identify the premises relevant in the current context and thereby reduce the branching factor of the proof search to a manageable size.
Revised: Thus, the main problem of reasoning in large theories is to identify the premises relevant in the current context and thereby reduce the branching factor of the proof search to a manageable size.

Original: This problem will become even more pronounced over time, as the theorem provers become more powerful, growing the number of available premises.
Revised: This problem will become even more pronounced over time, as the theorem provers become more powerful, growing the number of available premises.

Original: Previous works have relied on human proofs to either directly provide or learn (Bansal et al., 2019;
Revised: Previous works have relied on human proofs to either directly provide or learn (Bansal et al., 2019;

Original: Paliwal et al., 2020) which premises are relevant to the current proof.
Revised: Paliwal et al., 2020) which premises are relevant to the current proof.

Original: However, any open-ended system for mathematical reasoning needs to be able to learn which premises are relevant without human guidance.
Revised: However, any open-ended system for mathematical reasoning needs to be able to learn which premises are relevant without human guidance.

Original: In this work, we thus consider the problem of training a theorem prover without access to human proofs .
Revised: In this work, we thus consider the problem of training a theorem prover without access to human proofs .

Original: In particular, the contributions of this work are:
Revised: In particular, the contributions of this work are:

Original: We demonstrate training the theorem prover without human data can succeed when using deep reinforcement learning.
Revised: We demonstrate training the theorem prover without human data can succeed when using deep reinforcement learning.

Original: We do this with minimal additional engineering: by augmenting exploration of premises with a portion of the premises selected by a tf-idf (Manning et al., 2008) metric.
Revised: We do this with minimal additional engineering: by augmenting exploration of premises with a portion of the premises selected by a tf-idf (Manning et al., 2008) metric.

Original: We provide a ﬁrst side-by-side comparison of the effect of availability of human proofs on the ﬁnal theorem proving performance.
Revised: We provide a ﬁrst side-by-side comparison of the effect of availability of human proofs on the ﬁnal theorem proving performance.

Original: We learn to prove more theorems than the prover trained on human proofs alone and almost as many as with the combination of both approaches.
Revised: We learn to prove more theorems than the prover trained on human proofs alone and almost as many as with the combination of both approaches.

Original: We establish the underlying properties of the proof assistant and reinforcement learning setup that makes our approach work, by running multiple ablation experiments.
Revised: We establish the underlying properties of the proof assistant and reinforcement learning setup that makes our approach work, by running multiple ablation experiments.

Original: We thereby solve one of the road blocks on the way to open-ended learning of mathematical reasoning in large theories.
Revised: We thereby solve one of the road blocks on the way to open-ended learning of mathematical reasoning in large theories.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Reinforcement learning (RL) without imitation learning has been successful for computer games (cf.
Revised: Reinforcement learning (RL) without imitation learning has been successful for computer games (cf.

Original: Mnih et al.
Revised: Mnih et al.

Original: and it was demonstrated later in Silver et al.
Revised: and it was demonstrated later in Silver et al.

Original: (2017) that imitation learning is not necessary for complex games like Chess and Go.
Revised: (2017) that imitation learning is not necessary for complex games like Chess and Go.

Original: For more complex games with much larger action spaces, learning methods still rely on human imitation due to the exploration problem (cf.
Revised: For more complex games with much larger action spaces, learning methods still rely on human imitation due to the exploration problem (cf.

Original: Vinyals et al.
Revised: Vinyals et al.

Original: The question of exploration is well studied in reinforcement learning (Houthooft et al., 2016; Burda et al., 2019), but existing approaches such as (cid:15) -greedy do not work for premise selection because of very large (practically inﬁnite) action space.
Revised: The question of exploration is well studied in reinforcement learning (Houthooft et al., 2016; Burda et al., 2019), but existing approaches such as (cid:15) -greedy do not work for premise selection because of very large (practically inﬁnite) action space.

Original: We work in the setting of automating higher-order logic interactive theorem provers, since this is where there is most promise for building and formalizing large theories.
Revised: We work in the setting of automating higher-order logic interactive theorem provers, since this is where there is most promise for building and formalizing large theories.

Original: This is also evidenced by the fact that all large-scale formalization efforts by mathematicians have occurred in such systems (Gonthier, 2008; Hales et al., 2017).
Revised: This is also evidenced by the fact that all large-scale formalization efforts by mathematicians have occurred in such systems (Gonthier, 2008; Hales et al., 2017).

Original: Several works have explored RL for proof search in the context of connection provers (Färber et al., 2017; Kaliszyk et al., 2018; Zombori et al., 2019; 2020).
Revised: Several works have explored RL for proof search in the context of connection provers (Färber et al., 2017; Kaliszyk et al., 2018; Zombori et al., 2019; 2020).

Original: We are instead interested in addressing the issue of premise selection from a large knowledge base, through the use of deep reinforcement learning and without use of human proofs.
Revised: We are instead interested in addressing the issue of premise selection from a large knowledge base, through the use of deep reinforcement learning and without use of human proofs.

Original: This is the hard part of exploration due to the large repository of premises.
Revised: This is the hard part of exploration due to the large repository of premises.

Original: Premise selection itself has been an active research topic in the domain of automated theorem proving (Alama et al., 2014; Kaliszyk and Urban, 2015; Blanchette et al., 2016; Wang et al., 2017).
Revised: Premise selection itself has been an active research topic in the domain of automated theorem proving (Alama et al., 2014; Kaliszyk and Urban, 2015; Blanchette et al., 2016; Wang et al., 2017).

Original: Gauthier et al.
Revised: Gauthier et al.

Original: (2017) uses a tf-idf based premise selection model, but does not learn a model.
Revised: (2017) uses a tf-idf based premise selection model, but does not learn a model.

Original: Urban et al.
Revised: Urban et al.

Original: ; Kaliszyk et al.
Revised: ; Kaliszyk et al.

Original: ; Kaliszyk and Urban (2014); Piotrowski and Urban (2018) interleave runs of an automated theorem prover and a premise selection model using non-deep RL approaches.
Revised: ; Kaliszyk and Urban (2014); Piotrowski and Urban (2018) interleave runs of an automated theorem prover and a premise selection model using non-deep RL approaches.

Original: Deep learning has since signiﬁcantly improved the state of the art for premise selection, starting with Alemi et al.
Revised: Deep learning has since signiﬁcantly improved the state of the art for premise selection, starting with Alemi et al.

Original: , but these approaches have relied on human proofs.
Revised: , but these approaches have relied on human proofs.

Original: In our work, we use deep RL to learn premise selection while removing this dependence on human proofs.
Revised: In our work, we use deep RL to learn premise selection while removing this dependence on human proofs.

Original: We also provide a clear comparison of the effect of availability of human proofs to ﬁnal theorem proving performance, which has been lacking in the literature.
Revised: We also provide a clear comparison of the effect of availability of human proofs to ﬁnal theorem proving performance, which has been lacking in the literature.

Original: We use the HOList environment (Bansal et al., 2019) for HOL Light (Harrison, 1996).
Revised: We use the HOList environment (Bansal et al., 2019) for HOL Light (Harrison, 1996).

Original: Other ML environments for proof assistants include GamePad (Huang et al., 2019) and CoqGym (Yang and
Revised: Other ML environments for proof assistants include GamePad (Huang et al., 2019) and CoqGym (Yang and

Original: Deng, 2019) for Coq; and TacticToe (Gauthier et al., 2017) for HOL4 (Slind and Norrish, 2008).
Revised: Deng, 2019) for Coq; and TacticToe (Gauthier et al., 2017) for HOL4 (Slind and Norrish, 2008).

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Theorem proving.
Revised: Theorem proving.

Original: Proof assistants have been built to enable humans to write and then automatically check proofs.
Revised: Proof assistants have been built to enable humans to write and then automatically check proofs.

Original: In contrast to mathematical textbooks and papers, which are written mostly in natural language, we call mathematics formalized in proof assistants to be formal mathematics .
Revised: In contrast to mathematical textbooks and papers, which are written mostly in natural language, we call mathematics formalized in proof assistants to be formal mathematics .

Original: In this work we focus on the proof assistant HOL Light (Harrison, 1996), in which a wide range of mathematical theories have been formalized, and which has been famously used for the formalization of the Kepler conjecture (Hales et al., 2017).
Revised: In this work we focus on the proof assistant HOL Light (Harrison, 1996), in which a wide range of mathematical theories have been formalized, and which has been famously used for the formalization of the Kepler conjecture (Hales et al., 2017).

Original: HOL Light, as in many other proof assistants, relies mostly on “backward” proof steps.
Revised: HOL Light, as in many other proof assistants, relies mostly on “backward” proof steps.

Original: In contrast to “forward” proof steps, in which we only manipulate already proven statements, backward proofs start with a proof goal (the statement of the theorem to be proven) and apply proof tactics until all goals are proven.
Revised: In contrast to “forward” proof steps, in which we only manipulate already proven statements, backward proofs start with a proof goal (the statement of the theorem to be proven) and apply proof tactics until all goals are proven.

Original: In Figure 1, we give an example of a backward proof.
Revised: In Figure 1, we give an example of a backward proof.

Original: The goal here is to prove x + 0 = x , for all x ∈ N , and we apply the tactic MATCH_MP_TAC to the goal.
Revised: The goal here is to prove x + 0 = x , for all x ∈ N , and we apply the tactic MATCH_MP_TAC to the goal.

Original: Like many tactics, this tactic takes a
Revised: Like many tactics, this tactic takes a

Original: [Equation]
Revised: [Equation]

Original: ARITH_
Revised: ARITH_

Original: [Equation]
Revised: [Equation]

Original: PROVEN
Revised: PROVEN

Original: ARITH_TAC
Revised: ARITH_TAC

Original: PROVEN
Revised: PROVEN

Original: [Figure]
Revised: [Figure]

Original: premise (i.e. a previously proven theorem or lemma) as a parameter.
Revised: premise (i.e. a previously proven theorem or lemma) as a parameter.

Original: In this example, we use the induction theorem NAT_INDUCTION as a premise.
Revised: In this example, we use the induction theorem NAT_INDUCTION as a premise.

Original: This tactic application splits the ﬁrst goal into two subgoals, corresponding to the base case and the induction step.
Revised: This tactic application splits the ﬁrst goal into two subgoals, corresponding to the base case and the induction step.

Original: The semantics of an application of a proof tactic is that, if all subgoals are proven, then also the goal to which the tactic has been applied is proven.
Revised: The semantics of an application of a proof tactic is that, if all subgoals are proven, then also the goal to which the tactic has been applied is proven.

Original: In our case, we can prove both of the subgoals by simple arithmetic reasoning, provided by the tactic ARITH_TAC .
Revised: In our case, we can prove both of the subgoals by simple arithmetic reasoning, provided by the tactic ARITH_TAC .

Original: This tactic here does not require additional premises and returns an empty list of subgoals (for both of the subgoals we apply it to), meaning that they are proven, and hence the original goal is proven.
Revised: This tactic here does not require additional premises and returns an empty list of subgoals (for both of the subgoals we apply it to), meaning that they are proven, and hence the original goal is proven.

Original: Learning proof guidance for interactive theorem proving.
Revised: Learning proof guidance for interactive theorem proving.

Original: It is a long-standing goal in artiﬁcial intelligence to automate the theorem proving process described above, in particular to relieve the human experts from selecting the tactics and premises in each proof step.
Revised: It is a long-standing goal in artiﬁcial intelligence to automate the theorem proving process described above, in particular to relieve the human experts from selecting the tactics and premises in each proof step.

Original: Historically, most works focused on designing advanced search algorithms, leading to entire ﬁelds such as SAT and SMT solving and ﬁrst-order theorem proving.
Revised: Historically, most works focused on designing advanced search algorithms, leading to entire ﬁelds such as SAT and SMT solving and ﬁrst-order theorem proving.

Original: Recently, learning proof search strategies from data has become an area of active research (Alemi et al., 2016;
Revised: Recently, learning proof search strategies from data has become an area of active research (Alemi et al., 2016;

Original: Gauthier et al., 2017; Huang et al., 2019).
Revised: Gauthier et al., 2017; Huang et al., 2019).

Original: In this work, we follow the approach by Bansal et al.
Revised: In this work, we follow the approach by Bansal et al.

Original: (2019), which has shown the unique ability to ﬁnd relevant premises, which has been a big challenge for the classical techniques.
Revised: (2019), which has shown the unique ability to ﬁnd relevant premises, which has been a big challenge for the classical techniques.

Original: Figure 2 illustrates the tactic and premise selection architecture introduced by Bansal et al.
Revised: Figure 2 illustrates the tactic and premise selection architecture introduced by Bansal et al.

Original: and improved by Paliwal et al.
Revised: and improved by Paliwal et al.

Original: For each proof step, this architecture scores the tactics and it also produces a relevance score for each potential premise.
Revised: For each proof step, this architecture scores the tactics and it also produces a relevance score for each potential premise.

Original: Then, for each tactic, the top- k premises are given as arguments (unless the tactic does not require premise arguments).
Revised: Then, for each tactic, the top- k premises are given as arguments (unless the tactic does not require premise arguments).

Original: This results in a list of candidate tactic applications, which can be used as the actions in any search approach.
Revised: This results in a list of candidate tactic applications, which can be used as the actions in any search approach.

Original: We adopted the same search strategy as Bansal et al.
Revised: We adopted the same search strategy as Bansal et al.

Original: and Paliwal et al.
Revised: and Paliwal et al.

Original: (2020), which is a simple breadth-ﬁrst search with a parameter of how many of the candidate tactic applications should be expanded per proof goal.
Revised: (2020), which is a simple breadth-ﬁrst search with a parameter of how many of the candidate tactic applications should be expanded per proof goal.

Original: The tactic and premise selection architecture is trained on successful proofs.
Revised: The tactic and premise selection architecture is trained on successful proofs.

Original: For imitation learning, tuples of goal, tactic, and used premises are extracted from human proofs formalized in HOL
Revised: For imitation learning, tuples of goal, tactic, and used premises are extracted from human proofs formalized in HOL

Original: Light.
Revised: Light.

Original: The focus of this work, however, is to not learn from human proofs, and instead learn in a reinforcement learning setup from the proofs that earlier versions of the policy have found.
Revised: The focus of this work, however, is to not learn from human proofs, and instead learn in a reinforcement learning setup from the proofs that earlier versions of the policy have found.

Original:  L EARNING WITHOUT I MITATION
Revised:  L EARNING WITHOUT I MITATION

Original: In this section, we explain the setup for learning to prove in the absence of human proofs, and the considerations that informed our ﬁnal design.
Revised: In this section, we explain the setup for learning to prove in the absence of human proofs, and the considerations that informed our ﬁnal design.

Original: Much of mathematics that has been formalized by humans is in pursuit of formalizing certain theorems such as the four color theorem (Gonthier, 2008) and the Kepler conjecture (Hales et al., 2017).
Revised: Much of mathematics that has been formalized by humans is in pursuit of formalizing certain theorems such as the four color theorem (Gonthier, 2008) and the Kepler conjecture (Hales et al., 2017).

Original: Since formalization is a challenging and tedious process requiring experts, a very small fraction of mathematics is formalized after decades of human effort.
Revised: Since formalization is a challenging and tedious process requiring experts, a very small fraction of mathematics is formalized after decades of human effort.

Original: This work paves the way for a critical piece of a system wherein its knowledge base of formally proven theorems grows continuously.
Revised: This work paves the way for a critical piece of a system wherein its knowledge base of formally proven theorems grows continuously.

Original: We separate two key aspects of such a system.
Revised: We separate two key aspects of such a system.

Original: First, proposing potentially true statements, also known in the literature as conjecturing.
Revised: First, proposing potentially true statements, also known in the literature as conjecturing.

Original: Second, given a new statement, proving it without existing proofs to learn from.
Revised: Second, given a new statement, proving it without existing proofs to learn from.

Original: We would like to tackle the latter question in this work directly.
Revised: We would like to tackle the latter question in this work directly.

Original: The key information the human proofs provide is the overall direction of the proof via selection of the relevant premises at each proof step.
Revised: The key information the human proofs provide is the overall direction of the proof via selection of the relevant premises at each proof step.

Original: In case human proofs are available, one can ﬁrst train a machine learning model to imitate them (Section 3), as has been done in several previous works (Section 2).
Revised: In case human proofs are available, one can ﬁrst train a machine learning model to imitate them (Section 3), as has been done in several previous works (Section 2).

Original: [Figure]
Revised: [Figure]

Original: Reinforcement learning loop.
Revised: Reinforcement learning loop.

Original: In the absence of human proofs, we need a mechanism to incrementally improve a proof guidance model, which motivates the reinforcement learning setup we use.
Revised: In the absence of human proofs, we need a mechanism to incrementally improve a proof guidance model, which motivates the reinforcement learning setup we use.

Original: Figure 3a shows the components of the reinforcement learning loop from Bansal et al.
Revised: Figure 3a shows the components of the reinforcement learning loop from Bansal et al.

Original: (2019), which we build upon.
Revised: (2019), which we build upon.

Original: A proof guidance model is trained with continuously expanding training data.
Revised: A proof guidance model is trained with continuously expanding training data.

Original: In order to generate the continuously expanding training data, several theorem provers run in lockstep with training the policy and premise selection network.
Revised: In order to generate the continuously expanding training data, several theorem provers run in lockstep with training the policy and premise selection network.

Original: The provers try to prove the statements in the training set using the model for proof guidance as it is training.
Revised: The provers try to prove the statements in the training set using the model for proof guidance as it is training.

Original: If it manages to prove a statement, the proof is used to generate additional training data.
Revised: If it manages to prove a statement, the proof is used to generate additional training data.

Original: Intuitively, we would like to reward the choices of tactics and premises that were “useful” at each step in the proof.
Revised: Intuitively, we would like to reward the choices of tactics and premises that were “useful” at each step in the proof.

Original: A subtle but crucial aspect is that the proofs are pruned before generating the training data.
Revised: A subtle but crucial aspect is that the proofs are pruned before generating the training data.

Original: In this step, premises that are not necessary for the proof are removed.
Revised: In this step, premises that are not necessary for the proof are removed.

Original: This interplay between over-approximation and pruning is a major contributing factor to the efﬁciency of our exploration method and is studied in Subsection 4.2.
Revised: This interplay between over-approximation and pruning is a major contributing factor to the efﬁciency of our exploration method and is studied in Subsection 4.2.

Original: Figure 3b shows how the list of premises are picked, including our proposed change to aid exploration of the premises.
Revised: Figure 3b shows how the list of premises are picked, including our proposed change to aid exploration of the premises.

Original: Given a goal, the currently learnt model is used to pick the top- k 1 highest scored premises from the knowledge base of premises available: { P 1 , P 2 , . . .
Revised: Given a goal, the currently learnt model is used to pick the top- k 1 highest scored premises from the knowledge base of premises available: { P 1 , P 2 , . . .

Original: , P k 1 } .
Revised: , P k 1 } .

Original: Simultaneously, we propose generating another list of premises { Q 1 , Q 2 , . . .
Revised: Simultaneously, we propose generating another list of premises { Q 1 , Q 2 , . . .

Original: Q k 2 } to explore, picked according to a metric discussed shortly (Section 4.1).
Revised: Q k 2 } to explore, picked according to a metric discussed shortly (Section 4.1).

Original: The ﬁnal set of premises is obtained by interleaving the two premise lists.
Revised: The ﬁnal set of premises is obtained by interleaving the two premise lists.

Original: k 1 and k 2 are hyperparameters which can be varied to control exploitation of the learnt model (higher k 1 ) vs exploration (higher k 2 ).
Revised: k 1 and k 2 are hyperparameters which can be varied to control exploitation of the learnt model (higher k 1 ) vs exploration (higher k 2 ).

Original: The R EFERENCE setup is one without our modiﬁcation (i.e., k 2 = 0 ).
Revised: The R EFERENCE setup is one without our modiﬁcation (i.e., k 2 = 0 ).

Original: On the other hand, the E XPLORE setup includes additional premises as proposed here.
Revised: On the other hand, the E XPLORE setup includes additional premises as proposed here.

Original: There are three aspects to this that we wish to highlight, and that have informed the design: the strategy to generate the list of premises to explore, the effect of using irrelevant premises in an action (over-approximation of premise lists), and pruning.
Revised: There are three aspects to this that we wish to highlight, and that have informed the design: the strategy to generate the list of premises to explore, the effect of using irrelevant premises in an action (over-approximation of premise lists), and pruning.

Original: We discuss each of these in detail, designing experiments to inform different choices.
Revised: We discuss each of these in detail, designing experiments to inform different choices.

Original: 4. I NFORMATION RETRIEVAL FOR PREMISE SELECTION
Revised: 4. I NFORMATION RETRIEVAL FOR PREMISE SELECTION

Original: One of the key failure modes of our reinforcement learning setup trained without human proofs is not being able to prove new theorems.
Revised: One of the key failure modes of our reinforcement learning setup trained without human proofs is not being able to prove new theorems.

Original: With no new training data, the learning process would stall.
Revised: With no new training data, the learning process would stall.

Original: Thus, it is crucial that we continuously expand the horizon of theorems that we are able to prove.
Revised: Thus, it is crucial that we continuously expand the horizon of theorems that we are able to prove.

Original: Since we generate new training data only when we manage to prove a theorem, we wish to pick premises most likely relevant to prove the current goal.
Revised: Since we generate new training data only when we manage to prove a theorem, we wish to pick premises most likely relevant to prove the current goal.

Original: [Table]
Revised: [Table]

Original: In information retrieval literature, notions such as tf-idf (Manning et al., 2008) have been used to retrieve relevant documents corresponding to a query.
Revised: In information retrieval literature, notions such as tf-idf (Manning et al., 2008) have been used to retrieve relevant documents corresponding to a query.

Original: We view premise selection as a retrieval problem, thinking of the current goal we are trying to prove as the query, and the knowledge base of previously proven theorems (premises) as the documents from which we would like to retrieve.
Revised: We view premise selection as a retrieval problem, thinking of the current goal we are trying to prove as the query, and the knowledge base of previously proven theorems (premises) as the documents from which we would like to retrieve.

Original: Given a goal G , we use pre-engineered similarity scoring s ( G, P ) to rank a potential premise P for its usefulness in the next action (tactic application) with the target of proving the goal.
Revised: Given a goal G , we use pre-engineered similarity scoring s ( G, P ) to rank a potential premise P for its usefulness in the next action (tactic application) with the target of proving the goal.

Original: In our setup, we restrict our attention to functions of the form s ( G, P ) =
Revised: In our setup, we restrict our attention to functions of the form s ( G, P ) =

Original: (cid:104) r
Revised: (cid:104) r

Original: ( G ) / (cid:107) r ( G ) (cid:107) , r ( P ) / (cid:107) r ( P ) (cid:107)(cid:105) , where r is some simple vector representation of the expression and (cid:104)· , ·(cid:105) the dot product.
Revised: ( G ) / (cid:107) r ( G ) (cid:107) , r ( P ) / (cid:107) r ( P ) (cid:107)(cid:105) , where r is some simple vector representation of the expression and (cid:104)· , ·(cid:105) the dot product.

Original: This is also sometimes referred to as the cosine simarity.
Revised: This is also sometimes referred to as the cosine simarity.

Original: We consider r of the form r ( P ) i =
Revised: We consider r of the form r ( P ) i =

Original: tf ( P, i ) idf ( i ) , where the i -component corresponds to the tokens occurring in the formulas, idf is the “inverse document frequency” function which is precomputed by idf ( i ) =
Revised: tf ( P, i ) idf ( i ) , where the i -component corresponds to the tokens occurring in the formulas, idf is the “inverse document frequency” function which is precomputed by idf ( i ) =

Original: log( N/n i ) , where N is the total number of theorems and n i is the number of theorem containing the i -th token.
Revised: log( N/n i ) , where N is the total number of theorems and n i is the number of theorem containing the i -th token.

Original: For the term frequency function tf ( P, i ) , we have tested three possible choices: boolean weighting: 1 if f i ( P ) > 0 and 0 otherwise, logarithm weighting: 1 + log( f i ( P
Revised: For the term frequency function tf ( P, i ) , we have tested three possible choices: boolean weighting: 1 if f i ( P ) > 0 and 0 otherwise, logarithm weighting: 1 + log( f i ( P

Original: )) and natural: tf ( P, i ) =
Revised: )) and natural: tf ( P, i ) =

Original: f i ( P ) , where f i ( P ) is the number of occurrences of the i -th token in expression P .
Revised: f i ( P ) , where f i ( P ) is the number of occurrences of the i -th token in expression P .

Original: Running a full reinforcement learning loop uses over 25 years of CPU resources (Appendix B.2).
Revised: Running a full reinforcement learning loop uses over 25 years of CPU resources (Appendix B.2).

Original: Since it is prohibitive to run a lot of experiments with the full system, it suggests that we should ﬁrst evaluate the quality of similarity metrics in an ofﬂine manner.
Revised: Since it is prohibitive to run a lot of experiments with the full system, it suggests that we should ﬁrst evaluate the quality of similarity metrics in an ofﬂine manner.

Original: That is, we measure how well those metrics perform on existing proof traces and we only apply the similarity scores that performed best in a separate evaluation.
Revised: That is, we measure how well those metrics perform on existing proof traces and we only apply the similarity scores that performed best in a separate evaluation.

Original: To do so, we have adopted the metrics from Alemi et al.
Revised: To do so, we have adopted the metrics from Alemi et al.

Original: We have measured the average relative maximum rank and the top- k recall numbers for a small set of relevant k values (for k = 8 , 16 , 32 and 64 ) on a random selection of proofs comprising of 20% of the training set of the “complex” corpus of HOList.
Revised: We have measured the average relative maximum rank and the top- k recall numbers for a small set of relevant k values (for k = 8 , 16 , 32 and 64 ) on a random selection of proofs comprising of 20% of the training set of the “complex” corpus of HOList.

Original: The relative maximum rank of a true positive document (premise) is the absolute maximum rank of the true documents divided by the size of all possible premises (from which we make the selection), the average is then taken over all retrieval tasks.
Revised: The relative maximum rank of a true positive document (premise) is the absolute maximum rank of the true documents divided by the size of all possible premises (from which we make the selection), the average is then taken over all retrieval tasks.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: Note that low average maximum relative rank and high recall values indicate better premise selection performance of the similarity measure.
Revised: Note that low average maximum relative rank and high recall values indicate better premise selection performance of the similarity measure.

Original: To summarize, we pick the boolean term-frequency weighting scheme.
Revised: To summarize, we pick the boolean term-frequency weighting scheme.

Original: In addition, we speculate it could be helpful to add more variation of premises picked for exploration for a given goal, as over the course of a loop same goals are attempted multiple times.
Revised: In addition, we speculate it could be helpful to add more variation of premises picked for exploration for a given goal, as over the course of a loop same goals are attempted multiple times.

Original: To add this variation, we add a dropout probability hyperparameter p to components of the representation: r ( G ) i is zeroed with probability p when computing the representation for G .
Revised: To add this variation, we add a dropout probability hyperparameter p to components of the representation: r ( G ) i is zeroed with probability p when computing the representation for G .

Original: p is set to 0 .
Revised: p is set to 0 .

Original: 1 unless speciﬁed otherwise.
Revised: 1 unless speciﬁed otherwise.

Original: 4. O VER - APPROXIMATION OF PREMISES
Revised: 4. O VER - APPROXIMATION OF PREMISES

Original: Our exploration is based on the assumption that adding a few irrelevant premises does not inﬂuence the outcome of tactic applications signiﬁcantly.
Revised: Our exploration is based on the assumption that adding a few irrelevant premises does not inﬂuence the outcome of tactic applications signiﬁcantly.

Original: This allows us to accelerate the search by trying to over-approximate the set of premises used in tactic applications.
Revised: This allows us to accelerate the search by trying to over-approximate the set of premises used in tactic applications.

Original: We study the behavior of the most frequently occurring tactics that take premises as an argument: MESON and REWRITE .
Revised: We study the behavior of the most frequently occurring tactics that take premises as an argument: MESON and REWRITE .

Original: MESON is based on a ﬁrst-order logic solver.
Revised: MESON is based on a ﬁrst-order logic solver.

Original: We study how many extra premises we can add before
Revised: We study how many extra premises we can add before

Original: MESON starts to fail to prove a goal.
Revised: MESON starts to fail to prove a goal.

Original: To do so, we ﬁrst sample at random proof steps from the human proofs (which are always successful).
Revised: To do so, we ﬁrst sample at random proof steps from the human proofs (which are always successful).

Original: For each proof step we add as tactic arguments random irrelevant premises from the knowledge base of premises available at that proof step.
Revised: For each proof step we add as tactic arguments random irrelevant premises from the knowledge base of premises available at that proof step.

Original: We report the ratio of successful proof attempts after adding a set of tactic parameters with varying cardinality.
Revised: We report the ratio of successful proof attempts after adding a set of tactic parameters with varying cardinality.

Original: For
Revised: For

Original: REWRITE operations, we study the number of extra premises we can add and expect not to change the outcome of the rewrite operation.
Revised: REWRITE operations, we study the number of extra premises we can add and expect not to change the outcome of the rewrite operation.

Original: For both experiments, we sampled random proofs and one random proof step with the desired type of tactic ( MESON or REWRITE ), until we sampled 250 steps successfully.
Revised: For both experiments, we sampled random proofs and one random proof step with the desired type of tactic ( MESON or REWRITE ), until we sampled 250 steps successfully.

Original: Then for each l ∈ { 1 , 2 , 4 , 8 , 16 , 32 } , we sampled ﬁve different random parameter lists with length l .
Revised: Then for each l ∈ { 1 , 2 , 4 , 8 , 16 , 32 } , we sampled ﬁve different random parameter lists with length l .

Original: In our experiment, we append those parameters to our parameter list and execute the
Revised: In our experiment, we append those parameters to our parameter list and execute the

Original: [Table]
Revised: [Table]

Original: same tactic with the extended parameter list.
Revised: same tactic with the extended parameter list.

Original: Table 2 shows the ratio of application with the outcome being identical with that of the tactic application without the extra parameters.
Revised: Table 2 shows the ratio of application with the outcome being identical with that of the tactic application without the extra parameters.

Original: We can see that even adding 32 extra random parameters does not change the outcome of the rewrite tactics over 70% of the time.
Revised: We can see that even adding 32 extra random parameters does not change the outcome of the rewrite tactics over 70% of the time.

Original: However, MESON tends to time out with more than 32 extra premises.
Revised: However, MESON tends to time out with more than 32 extra premises.

Original: 4. P RUNING
Revised: 4. P RUNING

Original: As discussed in Section 4.2, if a tactic application succeeds, not all premises provided might have been used.
Revised: As discussed in Section 4.2, if a tactic application succeeds, not all premises provided might have been used.

Original: In fact, we are using this fact to accelerate our exploration.
Revised: In fact, we are using this fact to accelerate our exploration.

Original: However, we do not wish to learn from these irrelevant premises.
Revised: However, we do not wish to learn from these irrelevant premises.

Original: Thus, for generating training data, for each proof step in our proof we greedily try to remove the premises: given a proof step with premises { P i } n i =1 , we rerun the proof step without P n .
Revised: Thus, for generating training data, for each proof step in our proof we greedily try to remove the premises: given a proof step with premises { P i } n i =1 , we rerun the proof step without P n .

Original: If the result of the proof step remains unchanged, we drop P n .
Revised: If the result of the proof step remains unchanged, we drop P n .

Original: Then, we continue with trying to drop P n − 1 , and so on.
Revised: Then, we continue with trying to drop P n − 1 , and so on.

Original: We use the dropped premises as hard negatives, such that premises which are ranked highly by the model but are not useful are demoted.
Revised: We use the dropped premises as hard negatives, such that premises which are ranked highly by the model but are not useful are demoted.

Original: Demoting pruned premises allows other premises that had a high score but did not make it into the top- k to get a chance in the future.
Revised: Demoting pruned premises allows other premises that had a high score but did not make it into the top- k to get a chance in the future.

Original: Pruning also ensures that any extra premises added for exploration that are in fact unnecessary are not learnt upon.
Revised: Pruning also ensures that any extra premises added for exploration that are in fact unnecessary are not learnt upon.

Original:  E VALUATION
Revised:  E VALUATION

Original: Environment and benchmark.
Revised: Environment and benchmark.

Original: We evaluate our approach in the HOList environment (Bansal et al., 2019) based on the HOL Light proof assistant.
Revised: We evaluate our approach in the HOList environment (Bansal et al., 2019) based on the HOL Light proof assistant.

Original: We chose to use HOList because of the breadth of topics of mathematics in the dataset.
Revised: We chose to use HOList because of the breadth of topics of mathematics in the dataset.

Original: Additionally, HOList is already integrated into a reinforcement learning setup, which our approach relies on.
Revised: Additionally, HOList is already integrated into a reinforcement learning setup, which our approach relies on.

Original: We conduct our experiments on the “complex” corpus of the HOList benchmark derived from theorems in HOL Light’s mathematical library from various areas of mathematics such as topology, multivariate calculus, real and complex analysis, geometric algebra, and measure theory.
Revised: We conduct our experiments on the “complex” corpus of the HOList benchmark derived from theorems in HOL Light’s mathematical library from various areas of mathematics such as topology, multivariate calculus, real and complex analysis, geometric algebra, and measure theory.

Original: It includes well-known theorems such as Abel’s theorem for power series, the fundamental theorem of calculus, and that the roots of the characteristic polynomial of a complex matrix are its eigenvalues.
Revised: It includes well-known theorems such as Abel’s theorem for power series, the fundamental theorem of calculus, and that the roots of the characteristic polynomial of a complex matrix are its eigenvalues.

Original: Table 3 gives some addtional examples.
Revised: Table 3 gives some addtional examples.

Original: The task is to attempt to prove a theorem in the benchmark, with theorems and deﬁnitions appearing before it in the benchmark available as premises.
Revised: The task is to attempt to prove a theorem in the benchmark, with theorems and deﬁnitions appearing before it in the benchmark available as premises.

Original: The evaluation criterion is the fraction of theorems proven with this constraint.
Revised: The evaluation criterion is the fraction of theorems proven with this constraint.

Original: The benchmark comes with theorems divided into training, validation, and test sets.
Revised: The benchmark comes with theorems divided into training, validation, and test sets.

Original: Table 4 gives the statistics of theorems in the “complex” corpus of the benchmark.
Revised: Table 4 gives the statistics of theorems in the “complex” corpus of the benchmark.

Original: Deﬁnitions (totaling 637) and “core” corpus of theorems (totaling 2320), containing basic mathematics which the “complex” corpus builds upon, are available as premises, but are not attempted to be proven.
Revised: Deﬁnitions (totaling 637) and “core” corpus of theorems (totaling 2320), containing basic mathematics which the “complex” corpus builds upon, are available as premises, but are not attempted to be proven.

Original: Alternative
Revised: Alternative

Original: characterization of orthogonal matrices
Revised: characterization of orthogonal matrices

Original: [Table]
Revised: [Table]

Original: orth ( A ) ⇐⇒ ( ∀ i. |
Revised: orth ( A ) ⇐⇒ ( ∀ i. |

Original: A i | = 1 ∧
Revised: A i | = 1 ∧

Original: [Table]
Revised: [Table]

Original: i ! =
Revised: i ! =

Original: [Table]
Revised: [Table]

Original: ) Property
Revised: ) Property

Original: about absolute neighborhood retract
Revised: about absolute neighborhood retract

Original: (ANR).
Revised: (ANR).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Training and evaluation.
Revised: Training and evaluation.

Original: During training, we generate data for training by trying to prove statements in the training set of 10,214 theorems.
Revised: During training, we generate data for training by trying to prove statements in the training set of 10,214 theorems.

Original: We train for 8 million steps.
Revised: We train for 8 million steps.

Original: Details of our hardware setup and hyperparameters are in the Appendix.
Revised: Details of our hardware setup and hyperparameters are in the Appendix.

Original: For evaluation of all our experiments trained in the reinforcement learning setup, we focus on the number of statements proven in the held-out validation set of 3,225 theorems.
Revised: For evaluation of all our experiments trained in the reinforcement learning setup, we focus on the number of statements proven in the held-out validation set of 3,225 theorems.

Original: We run a continuous evaluation on samples of the validation set as well as a ﬁnal evaluation on the full validation set.
Revised: We run a continuous evaluation on samples of the validation set as well as a ﬁnal evaluation on the full validation set.

Original: These metrics are:
Revised: These metrics are:

Original: [Table]
Revised: [Table]

Original: Final
Revised: Final

Original: Cumulative validation validation
Revised: Cumulative validation validation

Original: Pure human imitation
Revised: Pure human imitation

Original: Bansal et al.
Revised: Bansal et al.

Original: 32.65% - Paliwal et al.
Revised: 32.65% - Paliwal et al.

Original: Human RL
Revised: Human RL

Original: Bansal et al.
Revised: Bansal et al.

Original: (2019) 38.9% not reported Human reference 59.5% 68.2% Human explore 59.9 % 69.1%
Revised: (2019) 38.9% not reported Human reference 59.5% 68.2% Human explore 59.9 % 69.1%

Original: Zero RL
Revised: Zero RL

Original: Zero reference 7.0% 
Revised: Zero reference 7.0% 7.3% Zero explore 56.3 %

Original: [Figure]
Revised: [Figure]

Original: .3% Zero explore 56.3 %

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: • Continuous validation performance (represented by dots in the plots) runs every 80,000 training steps on a random sample of validation theorems, and reports the fraction of proven theorems from that sample.
Revised: • Continuous validation performance (represented by dots in the plots) runs every 80,000 training steps on a random sample of validation theorems, and reports the fraction of proven theorems from that sample.

Original: Since not all validation theorems are attempted and the sample changes each evaluation, the metric is slightly noisy, but it allows us to monitor overﬁtting during training.
Revised: Since not all validation theorems are attempted and the sample changes each evaluation, the metric is slightly noisy, but it allows us to monitor overﬁtting during training.

Original: • Final validation performance (reported in the tables and plots) is the fraction of all validation theorems proven by the ﬁnal checkpoint at 8 million steps.
Revised: • Final validation performance (reported in the tables and plots) is the fraction of all validation theorems proven by the ﬁnal checkpoint at 8 million steps.

Original: This metric also allows for comparison with models trained purely by imitation learning.
Revised: This metric also allows for comparison with models trained purely by imitation learning.

Original: • Cumulative validation performance – reported in the tables and plots – is the fraction of all validation theorems proven by any continuous-validation run up until that point in the loop.
Revised: • Cumulative validation performance – reported in the tables and plots – is the fraction of all validation theorems proven by any continuous-validation run up until that point in the loop.

Original: The table reports the cumulative performance of the whole loop (i
Revised: The table reports the cumulative performance of the whole loop (i

Original: e., after 8 million steps
Revised: e., after 8 million steps

Original: Main experiment.
Revised: Hardware and hyperparameters. One of hyperparameters introduced is the number of additional premises picked by the model ( k 1 ) vs for additional exploration ( k 2 ). Since it is computationally expensive to run a single RL loop, we do not ﬁne-tune these. Rather than picking a ﬁxed value in a experiment, we make attempts over the course of a single experiment with different values to add more diversity.

Revised: In partcicular, k 2 is picked uniformily at random in the range [0 , 8] independently by each prover.

Revised: k 1 is then picked as k (total number of premises) minus k 2 .

Revised: For evaluation, to keep our numbers comparable we use the same prover timeout and maximum number of nodes explored as in Paliwal et al.

Revised: A full discussion about the hardware, hyperparameters and computational resources used can be found in the appendix.

Revised: Main experiment.

Original: In the main experiment we are interested in understanding the ability of the
Revised: In the main experiment we are interested in understanding the ability of the

Original: E XPLORE approach proposed in Section 4, in particular, its ability to learn to prove without human proof data available.
Revised: E XPLORE approach proposed in Section 4, in particular, its ability to learn to prove without human proof data available.

Original: This experiment is referred to as Z ERO E XPLORE .
Revised: This experiment is referred to as Z ERO E XPLORE .

Original: As we see in the plot on the right in Figure 4, the loop continuously expands the horizon of the theorems it is able to prove.
Revised: As we see in the plot on the right in Figure 4, the loop continuously expands the horizon of the theorems it is able to prove.

Original: To put the performance in context, we categorize the results on this benchmark into three categories.
Revised: To put the performance in context, we categorize the results on this benchmark into three categories.

Original: First, pure human imitation , wherein the model is trained only on human proof data and no theorem prover is used during training.
Revised: First, pure human imitation , wherein the model is trained only on human proof data and no theorem prover is used during training.

Original: Second, human RL , wherein the the model is trained on human proof data as well data generated by running the prover in a reinforcement learning (RL) loop.
Revised: Second, human RL , wherein the the model is trained on human proof data as well data generated by running the prover in a reinforcement learning (RL) loop.

Original: Finally, zero
Revised: Finally, zero

Original: RL , wherein no human data is available, and all data is generated by running a prover in an RL loop.
Revised: RL , wherein no human data is available, and all data is generated by running a prover in an RL loop.

Original: The results are summarized in the table in Figure 4.
Revised: The results are summarized in the table in Figure 4.

Original: Paliwal et al.
Revised: Paliwal et al.

Original: (2020) based on graph neural networks (GNN) is the state-of-the-art on pure human imitation on this benchmark, and we use the network architecture in our experiments as well.
Revised: (2020) based on graph neural networks (GNN) is the state-of-the-art on pure human imitation on this benchmark, and we use the network architecture in our experiments as well.

Original: Compared to pure human imitation, the Z ERO E XPLORE RL loop using no human proofs is able to prove more theorems on a single checkpoint: 49.95% (pure imitation) vs 56.3% (zero RL).
Revised: Compared to pure human imitation, the Z ERO E XPLORE RL loop using no human proofs is able to prove more theorems on a single checkpoint: 49.95% (pure imitation) vs 56.3% (zero RL).

Original: Next, we compare to the best human RL loop (H UMAN E XPLORE ), one with everything identical as in Z ERO E XPLORE , except we let the model learn from additional human proof data.
Revised: Next, we compare to the best human RL loop (H UMAN E XPLORE ), one with everything identical as in Z ERO E XPLORE , except we let the model learn from additional human proof data.

Original: We see that
Revised: We see that

Original: the
Revised: the

Original:  Z ERO E XPLORE loop comes very close to the performance of the corresponding human RL loop: 59.9% vs 56.3% on a single checkpoint, and 69.1% vs 64.1% cumulatively.
Revised: [Figure] Z ERO E XPLORE loop comes very close to the performance of the corresponding human RL loop: 59.9% vs 56.3% on a single checkpoint, and 69.1% vs 64.1% cumulatively.

Original: Z ERO E XPLORE is able to reach over 90% of the human RL loop’s performance.
Revised: Z ERO E XPLORE is able to reach over 90% of the human RL loop’s performance.

Original: This is an important indicator as H UMAN loops indicate a ceiling unrelated to availability of proof data: such as the proof search algorithm, or model architecture.
Revised: This is an important indicator as H UMAN loops indicate a ceiling unrelated to availability of proof data: such as the proof search algorithm [Figure] or model architecture.

Original: Finally, we compare against the RL setup from Bansal et al.
Revised: Finally, we compare against the RL setup from Bansal et al.

Original: and run it without human proof data (Z ERO R EFERENCE ).
Revised: and run it without human proof data (Z ERO R EFERENCE ).

Original: We run into the failure mode of not being able to prove new statements and thus stalling, discussed in Section 4.1.[Figure] [Figure] [Figure]
Revised: We run into the failure mode of not being able to prove new statements and thus stalling, discussed in Section 4.1.

Original:  A BLATION STUDIES
Revised:  A BLATION STUDIES

Original: Bootstrapping.
Revised: Bootstrapping.

Original: In Section 5, we observe that the Z ERO R EFERENCE loop stalls very quickly.
Revised: In Section 5, we observe that the Z ERO R EFERENCE loop stalls very quickly.

Original: Here, we try to understand to what extent is the failure a bootstrapping issue.
Revised: Here, we try to understand to what extent is the failure a bootstrapping issue.

Original: We attempt to prove all statements in the training set using the best hand-engineered metric in Section 4.1 for premise selection and random tactic selection.
Revised: We attempt to prove all statements in the training set using the best hand-engineered metric in Section 4.1 for premise selection and random tactic selection.

Original: This proves around 20% percent of the statements.
Revised: This proves around 20% percent of the statements.

Original: We start a zero RL loop as in Z ERO R EFERENCE , but providing these additional proofs to train upon, calling it
Revised: We start a zero RL loop as in Z ERO R EFERENCE , but providing these additional proofs to train upon, calling it

Original: Z ERO S EEDED .
Revised: Z ERO S EEDED .

Original: We see in Figure 5a that it does not stall like the reference loop.
Revised: We see in Figure 5a that it does not stall like the reference loop.

Original: At the same time, it does not reach the same level of performance as the Z ERO E XPLORE , which explores throughout.
Revised: At the same time, it does not reach the same level of performance as the Z ERO E XPLORE , which explores throughout.

Original: Premise selection ablation.
Revised: Premise selection ablation.

Original: Through this ablation, we wish to understand the capability of the hand-engineered metrics in Section 4.1 to prove theorems.
Revised: Through this ablation, we wish to understand the capability of the hand-engineered metrics in Section 4.1 to prove theorems.

Original: To keep the focus on premise selection, we still learn the tactic selection, and use similar amounts of resources trying to prove statements as in one RL loop.
Revised: To keep the focus on premise selection, we still learn the tactic selection, and use similar amounts of resources trying to prove statements as in one RL loop.

Original: Technically, we do this by setting k 1 (number of premises chosen by the learnt network) to 0 , and rest of the setup as in the Z ERO E XPLORE loop.
Revised: Technically, we do this by setting k 1 (number of premises chosen by the learnt network) to 0 , and rest of the setup as in the Z ERO E XPLORE loop.

Original: In Figure 5, under premise-selection ablation we see the results: it manages to prove 43% of the statements cumulatively, compared with 64% when learning with a combination of exploration and exploitation in the RL loop.
Revised: In Figure 5, under premise-selection ablation we see the results: it manages to prove 43% of the statements cumulatively, compared with 64% when learning with a combination of exploration and exploitation in the RL loop.

Original: Dropout.
Revised: Dropout.

Original: In Section 4.1 we suggest a 10% token dropout probability when deciding premises to explore to introduce more diversity of premises in the loop overall at the cost of picking slightly less relevant premises at a speciﬁc point.
Revised: In Section 4.1 we suggest a 10% token dropout probability when deciding premises to explore to introduce more diversity of premises in the loop overall at the cost of picking slightly less relevant premises at a speciﬁc point.

Original: We evaluate this experimentally (dropout ablation in Figure 5), but do not see a major difference: we can observe a very slight gain requiring further veriﬁcation.
Revised: We evaluate this experimentally (dropout ablation in Figure 5), but do not see a major difference: we can observe a very slight gain requiring further veriﬁcation.

Original: Human loop ablation.
Revised: Human loop ablation.

Original: We run a human loop where we do not add premises for exploration (human ablation, Figure 5).
Revised: We run a human loop where we do not add premises for exploration (human ablation, Figure 5).

Original: We do not see a signiﬁcant difference in the performance.
Revised: We do not see a signiﬁcant difference in the performance.

Original: It is not surprising, as the human loop has proofs for all statements and is thus not reliant on premise exploration to ﬁnd relevant premises, unlike the zero RL loops.
Revised: It is not surprising, as the human loop has proofs for all statements and is thus not reliant on premise exploration to ﬁnd relevant premises, unlike the zero RL loops.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we demonstrate that it is possible to learn a premise selection model for theorem proving in the absence of human proofs.
Revised: In this work we demonstrate that it is possible to learn a premise selection model for theorem proving in the absence of human proofs.

Original: We show that, on our benchmark, we exceed the performance of a network trained purely on human proofs, and approach the performance of the system that combines reinforcement learning with imitation learning.
Revised: We show that, on our benchmark we exceed the performance of a network trained purely on human proofs and approach the performance of the system that combines reinforcement learning with imitation learning

Original: A H YPERPARAMETERS A. P OLICY NETWORK TRAINING PARAMETERS
Revised: A H YPERPARAMETERS A. P OLICY NETWORK TRAINING PARAMETERS

Original: • batch size: 16 goals, 256 premises • number of workers:• optimizer: Adam • Adam epsilon: 1e- • initial learning rate: 1e- • learning rate decay: exponential, 0.98/100000 steps • embedding size:• non-linearity: ReLU • hidden layer dropout: 0.
Revised: • batch size: 16 goals, 256 premises • number of workers:• optimizer: Adam • Adam epsilon: 1e- • initial learning rate: 1e- • learning rate decay: exponential, 0.98/100000 steps • embedding size:• non-linearity: ReLU • hidden layer dropout: 0.

Original: • GNN hops:• layers per hop:• initializer range: 0.
Revised: • GNN hops:• layers per hop:• initializer range: 0.

Original: • pre-combiner embedding size:• number of combiner layers:• ratio of human training data: 0.7 (for human loops), 0.0 (for zero loops) • ratio of historical training data: 0.2 (for human loops), 0.5 (for zero loops) • ratio of fresh training data: 0.1 (for human loops), 0.5 (for zero loops)
Revised: • pre-combiner embedding size:• number of combiner layers:• ratio of human training data: 0.7 (for human loops), 0.0 (for zero loops) • ratio of historical training data: 0.2 (for human loops), 0.5 (for zero loops) • ratio of fresh training data: 0.1 (for human loops), 0.5 (for zero loops)

Original: A. P ROVER HYPERPARAMETERS
Revised: A. P ROVER HYPERPARAMETERS

Original: Some parameters are picked independently by each prover.
Revised: Some parameters are picked independently by each prover.

Original: These are picked uniformly at random from a given interval, are indicated below as [ n 1 , n 2 ] .
Revised: These are picked uniformly at random from a given interval, are indicated below as [ n 1 , n 2 ] .

Original: The intervals are inclusive of both end points.
Revised: The intervals are inclusive of both end points.

Original: Training provers:
Revised: Training provers:

Original: • total number of provers: 2000 (see continuous validation note) • training round interval (each time a model checkpoint is written out): 4000 training steps • prover tree search strategy:
Revised: • total number of provers: 2000 (see continuous validation note) • training round interval (each time a model checkpoint is written out): 4000 training steps • prover tree search strategy:

Original: BFS • timeout: 300 seconds • maximum number of actions considered (per goal): [10, 30] • maximum successful actions (per goal): [10, 18] • maximum number of premises ( k ): [2, 32].
Revised: BFS • timeout: 300 seconds • maximum number of actions considered (per goal): [10, 30] • maximum successful actions (per goal): [10, 18] • maximum number of premises ( k ): [2, 32].

Original: • number of premise samples per tactic:• tactic timeout: 500 milliseconds • maximum goals explored: 1000000 (practically, no limit)
Revised: • number of premise samples per tactic:• tactic timeout: 500 milliseconds • maximum goals explored: 1000000 (practically, no limit)

Original: Parameters that vary from experiment-to-experiment:
Revised: Parameters that vary from experiment-to-experiment:

Original: Loop name
Revised: Loop name

Original: Seed proofs
Revised: Seed proofs

Original: Premise selection
Revised: Premise selection

Original: Human
Revised: Human

Original: Generated
Revised: Generated

Original: Learnt model
Revised: Learnt model

Original: Exploration
Revised: Exploration

Original: Human reference
Revised: Human reference

Original: YesYes
Revised: YesYes

Original: No ( k 2 = 0 )
Revised: No ( k 2 = 0 )

Original: Human explore
Revised: Human explore

Original: Yes
Revised: Yes

Original: Exploration
Revised: Exploration

Original: Yes
Revised: Yes

Original: Yes ( k (cid:48) 2 ∈ [0 , 8] , p = 0 . 1 )
Revised: Yes ( k (cid:48) 2 ∈ [0 , 8] , p = 0 . 1 )

Original: Zero reference
Revised: Zero reference

Original: No
Revised: No

Original: Reference
Revised: Reference

Original: Yes
Revised: Yes

Original: No ( k 2 = 0 )
Revised: No ( k 2 = 0 )

Original: Zero explore
Revised: Zero explore

Original: No
Revised: No

Original: Exploration
Revised: Exploration

Original: Yes
Revised: Yes

Original: Yes ( k (cid:48) 2 ∈ [0 , 8] , p = 0 . 1 )
Revised: Yes ( k (cid:48) 2 ∈ [0 , 8] , p = 0 . 1 )

Original: Zero seeded
Revised: Zero seeded

Original: No
Revised: No

Original: Exploration
Revised: Exploration

Original: Yes
Revised: Yes

Original: No ( k 2 = 0 )
Revised: No ( k 2 = 0 )

Original: Zero hand-engineered only
Revised: Zero hand-engineered only

Original: No
Revised: No

Original: Exploration
Revised: Exploration

Original: No
Revised: No

Original: Yes ( k 2 = k , p = 0 . 1 )
Revised: Yes ( k 2 = k , p = 0 . 1 )

Original: Zero explore 0% dropout
Revised: Zero explore 0% dropout

Original: No
Revised: No

Original: Exploration
Revised: Exploration

Original: Yes
Revised: Yes

Original: Yes ( k (cid:48) 2 ∈ [0 , 8] , p = 0 . 0 )
Revised: Yes ( k (cid:48) 2 ∈ [0 , 8] , p = 0 . 0 )

Original: k 1 (number of premises to be picked by the network) and k 2 (number of premises to be added for exploration) are decided as follows.
Revised: k 1 (number of premises to be picked by the network) and k 2 (number of premises to be added for exploration) are decided as follows.

Original: In the table above, for experiments where k 2 is deﬁned k 2 is picked as shown.
Revised: In the table above, for experiments where k 2 is deﬁned k 2 is picked as shown.

Original: For experiments where k (cid:48) 2 is deﬁned, k 2 is derived it as max ( (cid:100) k/ 2 (cid:101) , k (cid:48) 2 ) .
Revised: For experiments where k (cid:48) 2 is deﬁned, k 2 is derived it as max ( (cid:100) k/ 2 (cid:101) , k (cid:48) 2 ) .

Original: k 1 is picked as k − k 2 .
Revised: k 1 is picked as k − k 2 .

Original: p is the token dropout probability, deﬁned in Section 4.1.
Revised: p is the token dropout probability, deﬁned in Section 4.1.

Original: For technical reasons, for the training to start some non-empty training data is needed.
Revised: For technical reasons, for the training to start some non-empty training data is needed.

Original: For human
Revised: For human

Original: RL loops, since human proof data is available, the generated seed data is not strictly necessary.
Revised: RL loops, since human proof data is available, the generated seed data is not strictly necessary.

Original: For zero RL loops, some data needs to be provided.
Revised: For zero RL loops, some data needs to be provided.

Original: We generate this data by trying to prove all theorems in the training set on a randomly initialized model (i.e. the 0-th checkpoint of a model).
Revised: We generate this data by trying to prove all theorems in the training set on a randomly initialized model (i.e. the 0-th checkpoint of a model).

Original: The hyperparameters used for the provers to generate the seed data is as follows:
Revised: The hyperparameters used for the provers to generate the seed data is as follows:

Original: • prover tree search strategy:
Revised: • prover tree search strategy:

Original: BFS • timeout: 1000 seconds • maximum number of actions considered (per goal):• maximum successful actions (per goal):• maximum number of premises: k = k 1 = 24 , k 2 = 

Revised: BFS • timeout: 1000 seconds • maximum number of actions considered (per goal):• maximum successful actions (per goal):• maximum number of premises: k = k 1 = 24 , k 2 =

Original: (reference), k = k 2 = 16 , k 1 = 0 , p = 0 . 

Revised: (reference), k = k 2 = 16 , k 1 = 0 , p = 0 .

Original: (exploration).
Revised: (exploration).

Original: • number of premise samples per tactic:• tactic timeout: 5000 milliseconds • maximum goals explored: 1000000 (practically, no limit)
Revised: • number of premise samples per tactic:• tactic timeout: 5000 milliseconds • maximum goals explored: 1000000 (practically, no limit)

Original: Validation provers: To keep this work comparable to prior results on HOList benchmark, we have veriﬁed that the timeout and maximum explored nodes are the same as in Paliwal et al.
Revised: Validation provers: To keep this work comparable to prior results on HOList benchmark, we have veriﬁed that the timeout and maximum explored nodes are the same as in Paliwal et al.

Original: We have also veriﬁed that the results in Bansal et al.
Revised: We have also veriﬁed that the results in Bansal et al.

Original: (2019) are within half a percent of the reported results with these timeout and maximum explored nodes.
Revised: (2019) are within half a percent of the reported results with these timeout and maximum explored nodes.

Original: • prover tree search strategy:
Revised: • prover tree search strategy:

Original: BFS • timeout: 1000 seconds • maximum number of actions considered (per goal):• maximum successful actions (per goal):• maximum number of premises:• number of premise samples per tactic:• tactic timeout: 500 milliseconds • maximum goals explored: 1000000 (practically, no limit)
Revised: BFS • timeout: 1000 seconds • maximum number of actions considered (per goal):• maximum successful actions (per goal):• maximum number of premises:• number of premise samples per tactic:• tactic timeout: 500 milliseconds • maximum goals explored: 1000000 (practically, no limit)

Original: Following only apply to continuous validation, not ﬁnal validation which is run at ﬁnal checkpoint, and on the full set.
Revised: Following only apply to continuous validation, not ﬁnal validation which is run at ﬁnal checkpoint, and on the full set.

Original: • continuous validation interval: 20 rounds (80,000 training steps) • probability to prove a validation theorem: 0.1 (Explanation: the 2000 provers running in the RL loop for training, each independently decides to re-purpose itself with this probability to help with continuous evaluation.
Revised: • continuous validation interval: 20 rounds (80,000 training steps) • probability to prove a validation theorem: 0.1 (Explanation: the 2000 provers running in the RL loop for training, each independently decides to re-purpose itself with this probability to help with continuous evaluation.

Original: With this probability, this leads to over 2000 proof attempts per validation interval on average.)
Revised: With this probability, this leads to over 2000 proof attempts per validation interval on average.)

Original: B C OMPUTATIONAL RESOURCE ANALYSIS B. H ARDWARE SETUP
Revised: B C OMPUTATIONAL RESOURCE ANALYSIS B. H ARDWARE SETUP

Original: We used eight NVIDIA Tesla V100 GPUs for distributed training, an additional GPU was used purely for evaluation, and we maintained a separate parameter server on a CPU machine.
Revised: We used eight NVIDIA Tesla V100 GPUs for distributed training, an additional GPU was used purely for evaluation, and we maintained a separate parameter server on a CPU machine.

Original: The provers generating training data and running validation run distributed, using 2000 CPUs.
Revised: The provers generating training data and running validation run distributed, using 2000 CPUs.

Original: B. A BSOLUTE RESOURCE USAGE ESTIMATE
Revised: B. A BSOLUTE RESOURCE USAGE ESTIMATE

Original: Here we estimate the resource usage of a reinforcement learning loop.
Revised: Here we estimate the resource usage of a reinforcement learning loop.

Original: Since the improvement of the premise guidance is heavily reliant on generation of data, we run up to 2000 theorem provers distributing the statements each prover is attempting to prove.
Revised: Since the improvement of the premise guidance is heavily reliant on generation of data, we run up to 2000 theorem provers distributing the statements each prover is attempting to prove.

Original: Computing predictions takes a few milliseconds but actions in the proof assistant can take up to half a second.
Revised: Computing predictions takes a few milliseconds but actions in the proof assistant can take up to half a second.

Original: We use 8 GPUs for training the policy network and the experience collection uses CPUs only.
Revised: We use 8 GPUs for training the policy network and the experience collection uses CPUs only.

Original: Combined with proof search, to have a reasonable chance of proving a statement, we run the theorem prover with a timeout of 5 minutes for a statement in the training set.
Revised: Combined with proof search, to have a reasonable chance of proving a statement, we run the theorem prover with a timeout of 5 minutes for a statement in the training set.

Original: Training over 5 days, a single reinforcement learning loop takes over 25 years of CPU resources and 960 hours of GPU resources.
Revised: Training over 5 days, a single reinforcement learning loop takes over 25 years of CPU resources and 960 hours of GPU resources.

Original: B. R ELATIVE RESOURCE USAGE ESTIMATES
Revised: B. R ELATIVE RESOURCE USAGE ESTIMATES

Original: For pure human imitation , there is no reinforcement learning, and thus is the least computationally intensive.
Revised: For pure human imitation , there is no reinforcement learning, and thus is the least computationally intensive.

Original: The models in Paliwal et al.
Revised: The models in Paliwal et al.

Original: (2020) were trained for 1 million training steps.
Revised: (2020) were trained for 1 million training steps.

Original: The best model took 26 hours to train with 8 GPUs, or around 200 GPU hours.
Revised: The best model took 26 hours to train with 8 GPUs, or around 200 GPU hours.

Original: The resource usage for reinforcement learning loops is estimated above (Appendix B.2).
Revised: The resource usage for reinforcement learning loops is estimated above (Appendix B.2).

Original: The resource usage comparison of explore versus reference loops is analyzed next, being the main change proposed in the work.
Revised: The resource usage comparison of explore versus reference loops is analyzed next, being the main change proposed in the work.

Original: The main difference is an additional ranking based on tf-idf.
Revised: The main difference is an additional ranking based on tf-idf.

Original: The time to rank based on tf-idf is very small compared to the ranking with the graph neural network.
Revised: The time to rank based on tf-idf is very small compared to the ranking with the graph neural network.

Original: Nevertheless, note that the overall timeout for proving is ﬁxed (5 minutes), and includes time to generate the actions as well as performing the actions in the environment.
Revised: Nevertheless, note that the overall timeout for proving is ﬁxed (5 minutes), and includes time to generate the actions as well as performing the actions in the environment.

Original: One way to compare training of these different models is to consider how long each model takes to reach a certain performance.
Revised: One way to compare training of these different models is to consider how long each model takes to reach a certain performance.

Original: For instance, we could try to reach performance of the best Paliwal et al.
Revised: For instance, we could try to reach performance of the best Paliwal et al.

Original: (2020) model, that of around 50% validation performance.
Revised: (2020) model, that of around 50% validation performance.

Original: This is a rough estimate, but the best
Revised: This is a rough estimate, but the best

Original: Human RL loop ( human explore ) reaches this performance after utilizing 68 hours of GPU resources and 2 years of CPU resources.
Revised: Human RL loop ( human explore ) reaches this performance after utilizing 68 hours of GPU resources and 2 years of CPU resources.

Original: On the other hand, the best Zero RL loop ( zero explore ) utilizes roughly 1.35 times the resources – 92 hours of GPU resources and 2.6 years of CPU resources.
Revised: On the other hand, the best Zero RL loop ( zero explore ) utilizes roughly 1.35 times the resources – 92 hours of GPU resources and 2.6 years of CPU resources.

Original: Alternatively, one could compare the performance after utilizing the same amount of training steps.
Revised: Alternatively, one could compare the performance after utilizing the same amount of training steps.

Original: For instance, say, after roughly 1 million steps which corresponds to roughly 120 hours of GPU resources and 3.4 years of CPU resources.
Revised: For instance, say, after roughly 1 million steps which corresponds to roughly 120 hours of GPU resources and 3.4 years of CPU resources.

Original: At this stage, the human explore is able to prove roughly 58% of the statements, whereas the zero explore is able to prove 55% of the statements.
Revised: At this stage, the human explore is able to prove roughly 58% of the statements, whereas the zero explore is able to prove 55% of the statements.

Original: Note that these numbers are for continuous validation at 1 million steps, so are bit approximate.
Revised: Note that these numbers are for continuous validation at 1 million steps, so are bit approximate.

Original: At the same point the cumulative validation performance is 63.5% and 57.2% for human explore and zero explore respectively.
Revised: At the same point the cumulative validation performance is 63.5% and 57.2% for human explore and zero explore respectively.

Revised: C HOL LANGUAGE FOR EXPRESSIONS

Revised: The following is not necessary to understand this paper, but given for the sake of completeness.

Revised: The data in the HOList benchmark extracted on HOL Light and given in the form of ‘S-expressions’.

Revised: This may be thought of as a serialized version of the abstract syntax tree of terms in HOL Light, given in preﬁx notation.

Revised: All terms have a well-deﬁned type , which is also contained in the S-expressions.

Revised: For instance, a boolean variable, would be given as (v bool x) .

Revised: Here v denotes that the term is a variable, bool denotes the type, and x denotes the name.

Revised: A variable named f denoting a function from boolean to boolean is represented as (v (fun (bool) (bool))

Revised: f) .

Revised: (a (v (fun (bool) (bool)) f) (v bool x) denotes f ( x ) , application of the function variable f to the variable x .

Revised: As a longer example, the second example in Table 3 about absolute neighbourhood retracts is represented as (a (c (fun (fun (fun (cart (real) N) (bool))

Revised: (bool))

Revised: (bool)) !)

Revised: (l (v (fun (cart (real)

Revised: N) (bool))

Revised: s) (a (a (c (fun (bool) (fun (bool) (bool))) =

Revised: =>) (a (c (fun (fun (cart (real) N) (bool))

Revised: (bool))

Revised: ANR)

Revised: (a (c (fun (fun

Revised: [Table]

Revised: cart

Revised: [Table]

Revised: real) N)

Revised: [Table]

Revised: [Table]

Revised: cart

Revised: [Table]

Revised: real) N)

Revised: [Table]

Revised: For a more detailed, yet gentle, introduction on HOL Light, we refer the reader to its tutorial (Harrison, 2011).