Paper: Lemur: Integrating Large Language Models in Automated Program Verification (ICLR-2024)

Reviewer comment (Theory):
This is a relatively minor comment but the formalization and the algorithm seem geared towards imperative languages where the notion of associating a property/invariant with a line number makes natural sense. It would be helpful to acknowledge that the proposed calculus might not necessarily be applicable to all programming languages.

Author response:
We will clarify that our approach is designed for imperative languages. Thank you for the clarification suggestion!

Reviewer comment (Novelty):
There has a large body of literature on data-driven techniques for learning loop invariants [1,2]. Unlike Code2Inv and Lemur, these past works use dynamic values of loop variables. Adding references to this body of work would help paint a fuller picture about this area.

Author response:
We will include references to these papers and discuss them in the 'Related Work' section. Thank you!

Reviewer comment (Experiments):
For the benchmark of difficult programs from SV-COMP, where are the placeholder lines added? Is it after every line in the program? If not, doesn't the location of the placeholder leak information, potentially aiding Lemur?

Author response:
We automatically insert placeholders at the beginning of each loop and right before the loop, which we deem a generally good heuristic in practice. We do not think we are leaking any information specific to the benchmarks to Lemur.

Reviewer comment (Theory):
It seems like the text and the formal expression for the second bullet point of Proposition 2.1 do not line up. Shouldn't the implication in the formal expression go the other way?

Author response:
Thank you for spotting that! Yes, it should be the other way in the formal expression.  We updated the paper.

Reviewer comment (Theory):
I don't think the last line on Page 2 is correct. A program without loops can have an unstable property because the program can still have multiple paths due to branches. Also, how can a property that is not an invariant be stable? Is this the case where stability is due to the property always being false?

Author response:
We say the property is stable, if within a single execution of the program, the property always evaluates to true or always evaluates to false. If there are no loops, the property will be evaluated at most once. This also holds when the program has branches, as only one path will be executed.

Reviewer comment (Reproducibility):
Is the time in Table 1 in seconds? When you say timeout of 1 hr or 10 minutes, is that the timeout for the entire dataset or each program?

Author response:
Yes, the time is in seconds.  And the timeout is per program.  
We will clarify that in the paper.

Reviewer comment (Presentation):
Does each bar in Figure 4 correspond to a single value for the number of proposals or a set of values? It seems like it is the latter case. It would help to clarify the figure.

Author response:
Each bar in Figure 4 corresponds to a set of values.  We will clarify this in the text.

Reviewer comment (Experiments):
My main concerns are related to the empirical evaluation of Lemur. The combination of LLMs and standard verification procedures is indeed promising, but I am left wondering how much the results are reliant on the specific LLM GPT4.

Author response:
**Q: How much is Lemur performance reliant on the invariant generator quality? How does Lemur fare without GPT4 as an invariant generator?**

We agree that this is an interesting question of how sensitive Lemur is to an LLM choice. We made a decision to build the first version of Lemur on GPT4 which are arguably the most powerful models and absorb the cost of using these models. 

*We ran additional experiments with the GPT-3.5 turbo model and updated the experimental evaluation in Tables 2a and 2b. We observe that GPT-4 outperforms GPT-3.5 turbo, especially on SV-COMP benchmarks. Based on these results, we hypothesize that the performance might drop if we use other open-source models (but this can change as these models are constantly improving).*

Reviewer comment (Reproducibility):
This is particularly important considering the economic cost of reproducing the experiment, which can be prohibitive for some.

Author response:
**Q: This is particularly important considering the economic cost of reproducing the experiment, which can be prohibitive for some.**

We agree with the reviewer that it might be costly to use. To be more concrete, we spent ~2 cents per call for GPT4. We chose to use the best LLM model out there to see what is the potential of such integration. Our total budget was less than $1000 for the project.

Reviewer comment (Experiments):
How much is Lemur performance reliant on the invariant generator quality?

Author response:
**Q: How much is Lemur performance reliant on the invariant generator quality? How does Lemur fare without GPT4 as an invariant generator?**

We agree that this is an interesting question of how sensitive Lemur is to an LLM choice. We made a decision to build the first version of Lemur on GPT4 which are arguably the most powerful models and absorb the cost of using these models. 

*We ran additional experiments with the GPT-3.5 turbo model and updated the experimental evaluation in Tables 2a and 2b. We observe that GPT-4 outperforms GPT-3.5 turbo, especially on SV-COMP benchmarks. Based on these results, we hypothesize that the performance might drop if we use other open-source models (but this can change as these models are constantly improving).*

Reviewer comment (Experiments):
How does Lemur fare without GPT4 as an invariant generator?

Author response:
**Q: How much is Lemur performance reliant on the invariant generator quality? How does Lemur fare without GPT4 as an invariant generator?**

We agree that this is an interesting question of how sensitive Lemur is to an LLM choice. We made a decision to build the first version of Lemur on GPT4 which are arguably the most powerful models and absorb the cost of using these models. 

*We ran additional experiments with the GPT-3.5 turbo model and updated the experimental evaluation in Tables 2a and 2b. We observe that GPT-4 outperforms GPT-3.5 turbo, especially on SV-COMP benchmarks. Based on these results, we hypothesize that the performance might drop if we use other open-source models (but this can change as these models are constantly improving).*

Reviewer comment (Experiments):
Can Lemur be used with non-LLM invariant generators such as Code2Inv?

Author response:
**Q: Can Lemur be used with non-LLM invariant generators such as Code2Inv?**

Code2Inv is trained to find an invariant at a fixed location in a program, which is not suitable for our framework.
Our framework requires an LLM oracle to be able to generate an invariant at any point in the program.

In principle, we can use any LLM and verification tools. For example, we added new results for GPT3.5 oracle instead GPT4 that are presented in the paper. As we mentioned above, our new results show that GPT-4 outperforms GPT-3.5 turbo on the tested benchmarks.

We are happy to run more experiments with open-source LLMs in the final version of the paper but we will not be able to provide these results during rebuttal, unfortunately.

Reviewer comment (Writing):
The derivation rules in Fig.1 are not 100% clear to me. What is the semantic of operator ": :"?

Author response:
**Minors Q: The derivation rules in Fig.1 are not 100\% clear to me. What is the semantic of operator ": :"?**

:: means that a list M is a concatenation of sublists, e.g. M = sublist1::sublist2

For example, we often use the following notation in Figure 1.
```
M= M'::p 
```
This means that a trail M is a concatenation of a trail M' and p, where p is the last element of M. We will further clarify this in the paper.

Reviewer comment (Presentation):
Figures 3 and 4 should have different (and self-explanatory) captions, i.e. explicitly mention the referred benchmarks.

Author response:
**Minors Q: Figures 3 and 4 should have different (and self-explanatory) captions, i.e. explicitly mention the referred benchmarks.**

Thanks, we will fix.

Reviewer comment (Evaluation):
"An important research question is whether modern AI models are capable of understanding ." .. indeed, but can you claim you do that?

Author response:
**Q: "An important research question is whether modern AI models are capable of understanding ." .. indeed, but can you claim you do that?**

It has been shown in [1] that LLMs have potential understanding programs. We take these ideas to a new level, showing that we can leverage this understanding to help state-of-the-art verification tools verify more C programs. So, we believe that we make a step toward showing AI understanding of realistic C programs.

[1] Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. Can large language models reason about program invariants? ICML23

Reviewer comment (Novelty):
Are you the first to ask LLMs to propose sub-goals?

Author response:
**Q: Are you the first to ask LLMs to propose sub-goals?**

Yes, we are the first to propose this idea.  We outlined our contributions in Introduction, we will emphasize that we are the first to propose these techniques.

Reviewer comment (Presentation):
I'd much prefer a better survey that would make your contributions clear, and a graphical description of the system that would allow me to position the different components.

Author response:
**I'd much prefer a better survey that would make your contributions clear, and a graphical descriotion of the system that would allow me to position the different componrrntd**

We will include a table that compares learning-based augmented verification techniques in Appendix E.

Reviewer comment (Theory):
"Finally, if the verifier V proves that the original property is not an invariant" -> is the verifier complete?

Author response:
**Q: " -Finally, if the verifier V proves that the original property is not an invariant" -> is the verifier complete?**

As discussed in the paper, V needs to be sound and we do not assume it is complete. For example, if the program deals only with bit-vectors, then the underlying procedure is complete. But if the program contains arrays, then the verification problem is undecidable and the verifier is incomplete.

Reviewer comment (Presentation):
Arguably the prompts are what makes everything else worthwhile. Yet, they are in appendix?

Author response:
**>? Arguabythe prompts are what makes everything else worthwhile. Yet, they are in appendix?**

For our framework, we have a small set of predefined template prompts that correspond to our rules, which is why they are included in the appendix. We kindly disagree with the reviewer that the prompts are key components in our framework. While we did put effort into designing these templates, the main component is the theoretically sound integration of LLMs with automated reasoners. If space permits, we are happy to move the prompt templates to the main text.

Reviewer comment (Theory):
It is good that the authors try to show that the proof rules are sound; however, the soundness proofs are simple and not very insightful. Indeed, finding a sound set of rules for simplifying/decomposing program verification is often not the difficult part of the process. The more technically challenging part is to have completeness results for some subclass of programs/properties. Unfortunately, the authors don't even try to do this.

Author response:
**Q: Completeness of calculus rules**

*We strongly disagree that the value of abstract calculus is only the proof of completeness*. 

Many highly impactful abstract calculus in SMT (e.g., notably Model-Constructing Satisfiability Calculus [1])  do not attempt to establish completeness results. In the context of formal methods, abstract calculus has the following benefits:
* A separation between the implementation and the theory,
* Highlighting the key features of the procedure in a modular manner,
* Providing formalized grounds for future extension.

[1] A Model-Constructing Satisfiability Calculus,  Leonardo de Moura & Dejan Jovanović 
 https://link.springer.com/chapter/10.1007/978-3-642-35873-9_1


Would the review agree that our calculus serves these purposes?

Reviewer comment (Experiments):
Since there are practical limits to prompt lengths for an LLM like GPT or GPT-4, this sets limits to how large a program can be verified using the proposed technique. This is highly undesirable. It would have been better if the authors attempted decomposition of the problem (perhaps guided by an LLM) and then applied the proposed technique to each decomposed part, and then stitched the results back together. Unfortunately, the authors completely avoid any such attempt/discussion.

Author response:
**Q: Scalability of verification**


As we mentioned in the "Discussion of Limitations" section, our approach currently works with smaller programs (<= 150 tokens after removing comments, unnecessary headers, and clang-formatting), and a decomposition approach might be good for scaling to larger programs, as the reviewer also suggested. 

Decomposing a large program into smaller pieces for verification purposes is known to be challenging. Using LLM to perform this task requires the representation of the program in some compact form, a specification language for pre-/post- conditions, and an efficient verification procedure to validate the correctness of those conditions.

Reviewer comment (Experiments):
SV-COMP has several tracks for verification. The total count of programs in SV-COMP is significantly larger than that reported by the authors. It is not clear whether some cherry-picking was done in selecting the examples.

Author response:
**Q: SV-COMP contains many, many more benchmarks. How does LEMUR compare with UAutomizer or other tools on the other benchmarks? Why are so few benchmarks chosen for comparison?**

We call UAutomizer/ESBMC first, before augmenting a verifier with GPT-based generated invariants. Hence, in theory, we can solve at least as many instances as UAutomizer can, provided there is an appropriate timeout for the initial call (Algorithm 1, line 5). 

However, our main goal was to demonstrate that GPT can aid UAutomizer/ESBMC in solving problems that *neither of these solvers can solve independently*. We selected 50 smaller benchmarks from reachability track that these tools cannot solve and demonstrated that GPT aid can help with 26 of these challenging benchmarks. We find it impressive that LLMs help to solve these benchmarks, even if we could only solve about half of them.

Reviewer comment (Evaluation):
An LLM like GPT-4 makes use of immense computational power at the server end to come up with proposals/suggestions quickly. It is not fair to discount this computational effort completely when comparing with the performance of other solvers that do not make LLM calls. As a consequence, I believe the time comparison is not fair.

Author response:
**Q:  An LLM like GPT-4 makes use of immense computational power at the server end to come up with proposals/suggestions fast. How do you propose to factor this in your comparisons for an apples-to-apples comparison. Neither ESBMC not UAutomizer were provided access to such computational power; so how is the comparison fair?**

*We do not see our work as a competitor to modern program verifiers. Our goal is to explore whether LLM can help these tools perform better.*

*Moreover, we point out that most existing verifiers (e.g., ESBMC and UAutomizer) are not designed to leverage immense computational power like GPUs or cloud computing, and our approach provides such means to empower the verifiers.*

Reviewer comment (Presentation):
What actual value does the set of rules provide to this work? Wouldn't it have sufficed to have Algorithm 1 directly, along with an explanation of the steps?

Author response:
**Q: What actual value does the set of rules provide to this work? Wouldn't it have sufficed to have Algorithm 1 directly, along with an explanation of the steps?**

Our derivation rules define an underlying proof calculus, while Algorithm 1 represents one of many possible strategies for applying these rules. We believe that the proof system is significant, as it formally defines the interaction between the verifier and oracles in a sound and modular way. Note that we designed the rules to have a *clear separation of responsibility between two powerful technologies* to take the best of both worlds. We also have introduced the notion of *stability* and a rule that leverages this new concept. We believe that we are the first to propose such formalization, which can be built upon in the future.

Reviewer comment (Evaluation):
For what kinds of <program, property> combinations do LLMs fare badly as far as suggesting assumptions is concerned? In some sense, this ought to depend on what kinds of data it has been trained on. I didn't see any discussion in this regard -- I would tend to think that there are <program, property> combinations for which LLMs will have a very difficult time generating good assumptions.

Author response:
**Q:  For what kinds of <program, property> combinations do LLMs fare badly as far as suggesting assumptions is concerned? In some sense, this ought to depend on what kinds of data it has been trained on. I didn't see any discussion in this regard -- I would tend to think that there are <program, property> combinations for which LLMs will have a very difficult time generating good assumptions.**

As shown in the experiment, there were 24 out of the 50 challenging benchmarks that we could not solve. Upon closer examination, we observed that it is challenging to 1)  generate complex invariants involving if-then-else logic; 2) generate invariants where a program contains multiple loops. We will expand the discussion of limitations section in the paper to clarify this point.

Reviewer comment (Experiments):
An LLM like GPT-4 makes use of immense computational power at the server end to come up with proposals/suggestions fast. How do you propose to factor this in your comparisons for an apples-to-apples comparison? Neither ESBMC nor UAutomizer were provided access to such computational power; so how is the comparison fair?

Author response:
**Q:  An LLM like GPT-4 makes use of immense computational power at the server end to come up with proposals/suggestions fast. How do you propose to factor this in your comparisons for an apples-to-apples comparison. Neither ESBMC not UAutomizer were provided access to such computational power; so how is the comparison fair?**

*We do not see our work as a competitor to modern program verifiers. Our goal is to explore whether LLM can help these tools perform better.*

*Moreover, we point out that most existing verifiers (e.g., ESBMC and UAutomizer) are not designed to leverage immense computational power like GPUs or cloud computing, and our approach provides such means to empower the verifiers.*