Paper: Decomposing the Enigma: Subgoal-based Demonstration Learning for Formal Theorem Proving (ICLR-2024)

Reviewer comment (Experiments):
I am curious if the diffusion model would collapse into generating a generic organization that works decently well with most statements.

Author response:
We deeply appreciate your insightful advice regarding the diffusion model's performance. Our empirical finding strongly suggests that there isn't a “one-size-fits-all” optimal demonstration organization. Specifically, even the most adaptable organization we identified could successfully prove only *3* distinct statements. Furthermore, we only observed *4* instances of such adaptable organizations on the miniF2F-test. This highlights the tailored nature of each organization to its corresponding statement. The limited scope of even the most adaptable organization indicates the challenge in identifying the optimal demonstration organization for each unique statement. This complexity further emphasizes the effectiveness of our model in efficiently searching for and determining the most suitable demonstration organization for individual statements. These discussions are now included in Appendix D of the updated manuscript.

Reviewer comment (Experiments):
It would also be nice to show some examples where the order of the demonstrations greatly affects the performances.

Author response:
We are sincerely grateful for your valuable suggestion and have taken steps to address it. We have added several illustrative examples in Figures 16-18 in Appendix E. These cases specifically highlight how the order of demonstrations can significantly impact the final performance.

Reviewer comment (Experiments):
Another interesting baseline to compare would be to search the optimal organization for each tested statement.

Author response:
Thank you for your constructive suggestion. We conducted additional experiments using 200 randomized organization sampling attempts for each statement in the miniF2F-valid and miniF2F-test to search for the optimal organization. The results are as follows:

| Method                       | Valid  | Test  |
|------------------------------|--------|-------|
| Ours                         | 48.0%  | 45.5% |
| Optimal Organization Search  | 48.4%  | 44.3% |

As noted in [1], the benefit of conducting extensive organization searches tends to diminish beyond 100 attempts. This finding highlights the inherent challenges presented by this dataset and indicates the effectiveness of our diffusion model in efficiently optimizing the search process, thereby conserving costs in using LLMs. These results have been incorporated into Appendix C.7 in the updated version of our manuscript.

Reviewer comment (Evaluation):
It seems that previous LLM-based algorithms do not utilize formal proof generation tools such as Sledgehammer. I am wondering if it's possible to add a comparison of the number of usages of such tools (or their resource consumptions), aligning with the LLM calls comparison.

Author response:
Thank you for your valuable feedback! We have further analyzed the average number of Sledgehammer calls and their execution times, comparing our method with the DSP algorithm[1]. This analysis was conducted on a machine equipped with 64 CPU cores.
It is important to note that DSP also uses Sledgehammer. We have clarified this point more explicitly in the updated version. The results are as follows, displaying the average number of Sledgehammer calls and their corresponding durations (in seconds) for each solved statement:

| **Method** | **Calls (valid)** | **Calls (test)** | **Duration (valid)** | **Duration (test)** |
|------------|-------------------|------------------|----------------------|---------------------|
| DSP        | 2.33              | 2.39             | 3.29                 | 2.98                |
| Ours       | 2.88              | 3.22             | 4.16                 | 4.94                |


These results show that our method has a slight increase in both the frequency of Sledgehammer calls and their execution times compared to DSP. Specifically, this increase is primarily observed in statements that our method can solve but DSP cannot. For these statements, the number of Sledgehammer calls on the miniF2F-valid and miniF2F-test are 3.21 and 5.38, respectively. This suggests that the need for Sledgehammer becomes more important as the problem complexity increases. These results have been incorporated into Appendix C.8 in the updated version of our manuscript.

These results indicate that optimizing the use of Sledgehammer could potentially enhance the effectiveness of existing LLM-based methods. However, it is important to note that such an optimization strategy falls beyond the scope of our current paper. Nonetheless, your suggestion has sparked interest in this as a valuable direction for future research.

Reviewer comment (Evaluation):
No std deviations or measurement of uncertainty in Tbl 1 & 2.

Author response:
Thank you for your valuable suggestions! We have included additional results with mean and standard deviation values to quantify the uncertainty associated with our method. Specifically, for the variants employing diffusion models (”Ours” and “-subgoal”), we trained three separate diffusion models as detailed in Section 3.4. For the variants without diffusion models (”-subgoal & diffusion” and “-diffusion”), we repeated the experiment three times. The results are as follows and we have also updated these values in Table 2 of our manuscript:

| Method               | Valid             | Test              |
|----------------------|-------------------|-------------------|
| Ours                 | 48.0% (*±0.4*)    | 45.5% (*±0.6*)    |
| -subgoal & diffusion | 41.4% (*±0.9*)    | 38.7% (*±1.2*)    |
| -subgoal             | 44.3% (*±0.7*)    | 40.6% (*±0.6*)    |
| -diffusion           | 46.9% (*±1.3*)    | 44.1% (*±0.9*)    |

These results demonstrate that the methods incorporating diffusion models exhibit relatively more stable performance.

For baseline methods in Table 1, we used reported results from their original papers to ensure a fair comparison, hence standard deviations for these are not included in our analysis.

Reviewer comment (Theory):
Why are words like "sorry" and "oops" used to determine that the proof is not valid? The provided justification is that these words indicate that the proof has been prematurely terminated, but I'm not quite sure I understand this justification.

Author response:
In the Isabelle proof assistant, "sorry" can be interpreted as "we presume the current proposition is true (without proving it)", and "oops" means that "we give up showing the current proposition". For example, both `lemma "1=(2::int)" sorry` and `lemma "1=(2::int)" oops` are valid Isabelle code: the first assumes the integers 1 and 2 are equal (and this is treated as a lemma that can then be used in other proofs); in the second case, we abandon the attempt to show 1=2. We, of course, will not accept either as a valid proof of the equation 1 = 2. Therefore, any proof that includes the terms "oops" or "sorry" will be deemed invalid.