Paper: A Language-Agent Approach to Formal Theorem-Proving (ICLR-2024)

Reviewer comment (Experiments):
The problem is formalised as an RL task, but then the authors say "In this paper, we do not take on this problem. Instead, we consider a fixed policy - a wrapper around a pretrained LLM (GPT-4) that can learn in-context - and show that this policy can achieve a high reward". See question below.

Author response:
**Ans.** Our approach is inspired by recent “verbal RL” approaches [1, 2] that use LLMs to perform control tasks (like game-playing) that were traditionally solved using RL. The big idea here is that unlike in traditional deep RL, there is no gradient-based policy update descent. Instead, RL-style exploration is combined with in-context learning: the agent interacts with the world, collects information, and then updates the LLM's prompt to elicit new behavior. 

The MDP formulation associates our approach with this perspective. The MDP captures **interactions** between our LLM and the external world (the proof environment), which form the primary difference between our approach and prior LLM-based approaches to theorem-proving. The sentence "Instead, we consider a fixed policy..." captures this essential difference between our method and classical RL approaches where one updates the policy iteratively.

Reviewer comment (Presentation):
The structure of the framework proposed is not very clear. The algorithm in Fig. 3 is quite high-level. The calls and interactions with the LLMs are not discussed in much detail.

Author response:
* Algorithm: We agree -- the algorithm (Figure 3) should definitely appear in the main body of the paper like it currently does. We were not proposing to move it but just offer additional details in the appendix. However, we agree that some of these details (such as how the use of the stack distinguishes us from prior methods) should appear in the main paper. We will make these changes in the revision.

Reviewer comment (Writing):
On p. 9, the authors say: "However, [COPRA] departs from these prior methods in using execution feedback and a more sophisticated search algorithm." It is not clear to me what this more sophisticate search algorithm is, specifically why it is sophisticate.

Author response:
**Ans.** The search algorithm maintains a stack and keeps track of previous failures, and current execution feedback. This information is then added back to the prompt helping the guidance model (GPT-4) to better direct the search. The algorithm also uses a partial order over goals (originally introduced in Proverbot) to ensure that tactics actually simplify the proofs. Most of the previous approaches tend to use a stateless best-first search strategy, but our search uses a form of stack-based DFS and is stateful (by backtracking when the model starts repeating the same failed steps for any given proof state). This not only optimizes the number of inference steps needed to find a proof but also lets us fail fast when the proof is too hard to find (see Tables 1 and 2 for comparison).

Reviewer comment (Theory):
Why defining the problem as an RL task, if this is not used in the methodology proposed?

Author response:
**Ans.** Our approach is inspired by recent “verbal RL” approaches [1, 2] that use LLMs to perform control tasks (like game-playing) that were traditionally solved using RL. The big idea here is that unlike in traditional deep RL, there is no gradient-based policy update descent. Instead, RL-style exploration is combined with in-context learning: the agent interacts with the world, collects information, and then updates the LLM's prompt to elicit new behavior. 

The MDP formulation associates our approach with this perspective. The MDP captures **interactions** between our LLM and the external world (the proof environment), which form the primary difference between our approach and prior LLM-based approaches to theorem-proving. The sentence "Instead, we consider a fixed policy..." captures this essential difference between our method and classical RL approaches where one updates the policy iteratively.

Reviewer comment (Novelty):
What is sophisticate about the search algorithm used in COPRA?

Author response:
**Ans.** The search algorithm maintains a stack and keeps track of previous failures, and current execution feedback. This information is then added back to the prompt helping the guidance model (GPT-4) to better direct the search. The algorithm also uses a partial order over goals (originally introduced in Proverbot) to ensure that tactics actually simplify the proofs. Most of the previous approaches tend to use a stateless best-first search strategy, but our search uses a form of stack-based DFS and is stateful (by backtracking when the model starts repeating the same failed steps for any given proof state). This not only optimizes the number of inference steps needed to find a proof but also lets us fail fast when the proof is too hard to find (see Tables 1 and 2 for comparison).

Reviewer comment (Evaluation):
It is unfair to compare the number of inferences made with REPROVER in Figure 5, as COPRA utilizes GPT-4 to prove theorems, while REPROVER employs a much smaller LLM. One query from GPT-4 is far more powerful than a single inference from REPROVER's LLM.

Author response:
Our objective for comparing the number of inferences is as follows:

a. We want to test the hypothesis of whether one can match the performance of fine-tuned models using just in-context learning. There is a recent trend of using Language Agent [2, 3] for performing control tasks using in-context “verbal RL”. We wanted to explore this idea in the context of theorem proving. It is important to note that ReProver is finetuned on proof-state and tactic pair data, but our approach is completely in-context. 

b. The intention behind pass@k-inferences is to assess the speed of the proposed method and the effectiveness of the LLM or neural network to guide the proof search. To avoid any confusion we plan to rename this metric as pass@k-guidance budget, which is a reasonable metric as it does the right trade-off in accounting for the time taken to complete a proof and at the same time ignores very low-level hardware details. Even if ReProver uses a smaller model, it can make a lot more inferences in a given time. However, we observe that ReProver is slower in finding proofs even with the wall clock time. This means that the effectiveness of the guidance model is important in quickly finding the proof. We think that Coq/Lean ITP run time contributes more significantly when the guidance model is not effective search-wise (i.e. predicts wrong tactics). We can present an argument for ReProver as to why it is slower. We know that on average the time taken per inference (which includes time to run on ITP as well) is around 1.55 seconds for ReProver and 6.73 seconds for Copra (From Table 2). Even if we assume ReProver doesn’t take any time (zero time) to run the inferences and spends all of 1.55 seconds running the tactic on ITP, then we can assume that about 5 seconds (6.73 - 1.55) is the average response time for GPT-4. However, we see that the number of inferences used by ReProver is about 46x higher on success. This interestingly shows up in the wall clock time too which is around 9x higher (~46x/5) for ReProver on success, so there is a tradeoff between the two, but the number of inferences dominates when the guidance model is not good. So, if the guidance model is good (it may be as big as GPT), we can empirically argue that asymptotically the search will converge to proof faster (given that it can be found using that guidance model).

Reviewer comment (Novelty):
The authors seem to overstate their claim of having 'the first LLM-agent approach' with 'state-of-the-art' performance.

Author response:
Thank you for this point. We will make our claim of being state-of-the-art more nuanced, as we recognize that there are many versions of the state-of-the-art depending on how the problem is formulated (for example, whether the use of informal proofs is allowed or not).

Reviewer comment (Evaluation):
Can you explain the performance gap mentioned in point 1 of the weaknesses?

Author response:
“Decomposing the Enigma” [1] is based on Draft-Sketch-Proof (DSP) [4] style approaches, which uses informal proofs to find formal proofs.
Having access to informal proofs (human-written or LLM-generated) simplifies the problem of synthesizing the formal proof into more of a translation problem, and that is one of the reasons why DSP-like approaches perform well on miniF2F datasets. It is important to note that these numbers are not comparable across different languages, [1] uses Isabelle which has powerful automatic reasoning tools like Sledgehammer, unlike Lean. As we mentioned in the related work section of our paper, this idea is orthogonal to the ideas in our paper, and we will explore ways to combine it with our methods in future work. 

That said, we also note that informal proofs are hard to get in real-world mathematical reasoning settings such as software verification like [CompCert](https://en.wikipedia.org/wiki/CompCert). Also, the approach prescribed in [1] starts with a manually written subgoal-based proof, and then subsequently refines it. By contrast, our approach is completely automatic. 

We agree that COPRA doesn’t outperform Proverbot on CompCert dataset in absolute numbers, but it uses much fewer inferences to find those proofs. It is quite possible that COPRA could do more proofs if we allowed more inference steps, but we are limited by our cost and API budget for GPT-4.

Reviewer comment (Theory):
How can it be verified that theorems are proved sufficiently?

Author response:
COPRA generates formal proofs. Formal proofs are machine-checkable and Interactive Theorem Provers (ITP) like Lean are used for the purpose of verifying the correctness of [formal proofs](https://en.wikipedia.org/wiki/Formal_proof). Often formal proofs are much more rigorous and pedantic than informal proofs.

Reviewer comment (Novelty):
Why is COPRA claimed to be "the first LLM-agent approach to formal theorem-proving" when previous works like REPROVER and Decomposing the Enigma [1] might also be considered as LLM-agent approaches?

Author response:
Proof agents, like other Language Agents, need to have more human-like interactions with the interactive theorem prover (ITP) and generate proof step-by-step along with using external knowledge and tools. In approaches like ReProver, Baldur, PACT, etc., the model is not trained to interact with ITP. Instead, it is trained to either generate the whole proof all at once or one proof step at a time. Also, there is no use of execution feedback, history of proof so far, or known failed proof steps.

Specifically, in the case of ReProver, there is no use of error feedback from ITP. Most of the current work uses a guidance model to carry out a search for completing the proof in a step-by-step manner. The guidance model is trained to predict the next step and the search is carried out using the best-first strategy. Some other methods generate the whole proof in one shot. Other approaches use informal proofs, for example in (Draft-Sketch-Proof) DSP [4].  While DSP might be using external knowledge (like generating informal proofs, hammer), there is no interaction with the ITP. The proof sketch is generated all at once and then other symbolic techniques (like hammers) are used to fill these holes. Also, in applications like software verification (e.g. our CompCert benchmark), the notion of informal proof is not well defined and there is no informal statement to begin with (which is required for approaches like DSP). 

Language Agents are used to perform control tasks (naturally framed using an MDP) through “verbal RL” [2, 3]. This is different from traditional deep RL because there is no gradient descent. Instead, the method synthesizes RL-style exploration with  in-context learning. Neither ReProver nor “Decomposing the Enigma” frame theorem-proving as a control/MDP task.

Reviewer comment (Evaluation):
What is the average API cost for COPRA per proven theorem?

Author response:
It depends on the number of inferences needed, the model used, and the length of the proof. It is somewhere from `$0.75-$5` per theorem on miniF2F and CompCert datasets.

Reviewer comment (Novelty):
Novelty. It seems that the method is similar to retrieval-based LLM in the sense that the policy uses an external database. It would be great if the authors could compare with this line of works in detail, especially the ReProver paper.

Author response:
1. Questions about novelty and differences from existing retrieval-based LLM approaches like ReProver.  

**Ans.** Our approach is based on the new paradigm of Language Agents, where one uses LLMs to perform control tasks and interact with and external world. In particular, we were inspired by Reflexion [2], which uses language agents to solve natural language RL tasks, and Voyager [3], which uses language agents to play Minecraft. This approach is different from traditional deep RL because there is no gradient-based policy update. Instead, RL-style exploration is combined with in-context learning. 

In the proof synthesis setting, an LLM agent needs to have human-like interactions with the underlying interactive theorem prover (ITP) and generate a proof step-by-step along with using external knowledge and tools. In approaches like ReProver, Baldur, PACT, etc., the model is not trained to interact with the ITP. The proof steps are either generated all at once or one at a time, but there is no use of execution feedback, history of proof so far, or known failed proof steps. 

Specifically, in the case of ReProver, there is no use of error feedback from ITP. Most of the current work uses a guidance model to carry out a search for completing the proof in a step-by-step manner. The guidance model is trained to predict the next step and the search is carried out using the best-first strategy. Some other methods generate the whole proof in one shot. Other approaches use informal proofs, for example in Draft-Sketch-Proof (DSP) [4].  While DSP might be using external knowledge (like generating informal proofs, hammer), there is no interaction with the ITP. The proof sketch is generated all at once and then a powerful automatic reasoning tool (like hammer) is used to fill the holes. Also, it is important to note that in certain situations like software verification (e.g. CompCert) the notion of informal proof is not well defined and there is no informal statement to begin with (which is required for approaches like DSP).

Reviewer comment (Evaluation):
It seems that the proposed method does not significantly outperform ReProver.

Author response:
2. “It seems that the proposed method does not significantly outperform ReProver.”

**Ans.** It is true that the set of theorems that COPRA eventually discovers is not much larger than what Reprover discovers. However, COPRA  finds the proofs much faster than ReProver at least 9x faster in absolute wall clock time. COPRA also fails faster when it is not able to find the proofs. We weren't able to run COPRA on a very large number of timesteps because of the high cost of GPT-4 queries; however, we expect that doing so would improve COPRA's performance even more. 

More generally, we believe that as foundation models get more and more capable and cheaper (think of OpenAI's recent announcement regarding GPT-4-turbo, which significantly boosts GPT-4's context window while slashing costs), the benefits of COPRA-like approaches will become even clearer.

Reviewer comment (Experiments):
Ablation studies might be needed to understand the role played by the RL component.

Author response:
3. “Ablation studies might be needed to understand the role played by the RL component.”

**Ans.** Please let us know any specific ablations that you would want to see, and we would be happy to run them.

Reviewer comment (Evaluation):
Is there a difference between formal and informal theorem proving? It seems that there are some recent works that this work (LYRA: ORCHESTRATING DUAL CORRECTION IN AUTOMATED THEOREM PROVING) has a much higher score on miniF2F.

Author response:
4. “Is there a difference between formal and informal theorem proving? It seems that there are some recent works that this work (LYRA: ORCHESTRATING DUAL CORRECTION IN AUTOMATED THEOREM PROVING) [1] has a much higher score on miniF2F.”

**Ans.** A formal proof is something that can be automatically verified using an Interactive Theorem Prover (ITP), whereas an informal proof can only be verified by a human. ITP is a software tool to assist with the development of formal proofs by human-machine collaboration. This involves some sort of interactive proof editor, or other interfaces, with which a human can guide the search for proofs. Often formal proofs are much more rigorous and pedantic than informal proofs. So informal proof can be loosely considered as a proof sketch based on which one can write rigorous machine-checkable formal proofs.

The LYRA [1] paper was announced after the ICLR deadline; however, we will be happy to cite this work in our final version. It uses the DSP [4] pipeline which uses informal proofs to find the formal proofs. This works very well on datasets like miniF2F which have problems from math competitions. However, real-world math formulations are not necessarily math competition problems with well-known informal proofs. Certain domains like software verification like [CompCert](https://en.wikipedia.org/wiki/CompCert) don’t have any notion of informal proofs. It is important to note that having access to informal proofs (human-written or LLM-generated) simplifies the problem of synthesizing the formal proof into more of a translation problem, and that is one of the reasons why DSP-like approaches perform well on miniF2F dataset. These approaches will work very well if the LLM can use its memory component to effectively recall the informal proof of a well-known math competition problem. However, a lot of formal theorem proving happens on a completely new set of formalizations which are heavily customized depending on the domain. For example, in the field of software verification, we will have a custom mathematical model defined for the software which has no informal proofs or definitions which can be recalled from memory by the LLMs. It is also important to note that the accuracy numbers in [1] are not directly comparable to ours because they use different proof languages. Isabelle is used in [1, 4] which has powerful automatic reasoning tools like Sledgehammer, unlike Lean.

Our approach is generic and can work for any domain where formal theorem proving is used. It is one of the first methods which is domain and proof language agnostic.

Reviewer comment (Theory):
What is the role played by the RL part? I understand that you formulate the problem as an MDP and then the language-agent essentially mimics an RL algorithm. What is this particular RL algorithm? How to handle exploration-exploitation tradeoff?

Author response:
5. “What is the role played by the RL part? I understand that you formulate the problem as an MDP and then the language-agent essentially mimics an RL algorithm. What is this particular RL algorithm? How to handle exploration-exploitation tradeoff?”

**Ans.** Framing theorem proving as an MDP allows us to use the recent “verbal RL” techniques [2, 3] which use LLM for solving some natural language RL problems. This is different from traditional RL because there is no gradient descent, rather in-context Learning in LLM which resembles more like an exploration-only RL policy.

Reviewer comment (Experiments):
Is RL really essential here? Can you replace it with other planning methods such as tree search, or even a close-loop planning method such as ReAct?

Author response:
6. “Is RL really essential here? Can you replace it with other planning methods such as tree search, or even a close-loop planning method such as ReAct?”

**Ans.** Learning from the environment (Interactive Theorem Prover) feedback is very important in fixing issues with the proof steps as one can see in the Figure 4 example. It is an interesting example that shows how keeping track of failures helps in fixing the error which cannot be achieved using a simple tree search. We can say that empirically, changing the prompt leads makes the policy explore different proof steps nudging the proof search in the right direction.
It is also important to note that knowing previously failed proof steps helps in nudging the LLM to look for alternatives that will work or pursue a different direction of search.