Paper: DS-Prover: A Dynamic Sampling Based Approach for Neural Theorem Proving (ICLR-2024)

Reviewer comment (Evaluation):
Data augmentation: In table 1, one can actually see that the data augmentation helps for problems in miniF2F, but is actually detrimental to performance for problems in ProofNet (which are harder) and problems in Mathlib (which cover wider mathematical domains). Therefore, the conclusion ought to be that the data augmentation might improve things for certain problems, but cannot increase performance across the board or in general.

Author response:
**W1:** Thanks for pointing this out, we now have improved our statement.

Reviewer comment (Evaluation):
(Critical) Misleading claim: I'm surprised that the authors mentioned the HTPS paper by Lample et al., but not their results. The HTPS paper achieved a success rate of >40\% on the miniF2F with pass@64, compared to 30\% in this paper with pass@1. Of course, the success rates between pass@1 and pass@64 are very different, but one should be very careful before making the claim **We achieve a new state-of-the-art performance of 30.6% on MiniF2F using Lean** if its success rate is 10\% (absolute), or 25\% (relative) lower than a paper published 17 months ago. Some experiments for DS-solver at a higher pass@k should be performed before such claims can be verified.

Author response:
**W6:** Thank you for pointing this out. We acknowledge that our best performance is only for pass@1, as we did not conduct experiments for higher pass@k. We will correct our statement accordingly.

Reviewer comment (Reproducibility):
For how many steps was the model trained for? What are the training and validation metrics? What is the experimental wallclock time limit per problem for miniF2F, ProofNet and Mathlib?

Author response:
**Q1:** For early stopping, we used the Pass@1 of the validation dataset. After each 0.5 epoch in the augmented data model (1 epoch in the original data model) we used the current model to attempt to prove the theorems in the validation dataset. And we selected the model with the best Pass@1 on the validation data for our final evaluation.  

We had given a time limit of 10 minutes to find proof for all the theorems in each dataset.

Reviewer comment (Evaluation):
The improvement by the dynamic sampling seems incremental. The "Original data" columns in Table 1 shows only the improvement of 0.4 points against the optimized LeanDojo.

Author response:
Although the increment in performance may be smaller in some cases, we can also observe that dynamic sampling consistently either outperforms fixed sampling or performs equally well, never exhibiting inferior performance. This consistency makes dynamic sampling a better choice of algorithm for searching proofs within a given time limit.

Reviewer comment (Experiments):
The effect of the data augmentation is not entirely clear because there is no experiment that employs only it.

Author response:
Using only the augmented data would result in longer proofs since all the generated tactics will only have a maximum of one premise. Consequently, we would need to search the tree in greater depth to find proof, which would increase the time required for the search. Therefore, we decided not to perform this experiment.

Reviewer comment (Experiments):
Not all the experimental settings are clear. Specifically, I cannot find how many tactics are sampled in the fixed sampling.

Author response:
Thanks for pointing this out. In the fixed sampling method, we sampled 64 tactics each time. We have now included this information in the paper.

Reviewer comment (Writing):
The paper cites other works without parenthesizing the author names. It lowers the readability.

Author response:
Thank you for pointing this out, we now have corrected the citations in the paper.

Reviewer comment (Experiments):
How many tactics are sampled in the fixed sampling? Does changing the number of sample tactics influence the result?

Author response:
In fixed sampling, 64 tactics are sampled each time. 

Yes, decreasing the number of tactics to sample and apply as time passes influences the result. We have added new results discussing this aspect in the paper.

Reviewer comment (Reproducibility):
Not clear from the paper, if model weights and code will be open-sourced

Author response:
Yes, we plan to make the Model, Code, and DataSets available as open-source.

Reviewer comment (Experiments):
Interesting ablations and more in-depth analysis is missing (more detailed analysis of time tradeoff in Figure 2)

Author response:
We've conducted additional experiments within this context and intend to incorporate these findings into our paper.

Reviewer comment (Novelty):
The related work is highly insufficiently discussed

Author response:
Thank you for bringing up this point. We're enhancing our related works section by including further discussions on papers related to theorem provers beyond Lean.

Reviewer comment (Reproducibility):
Will the code and models be open-source?

Author response:
Yes, we plan to make the Model, Code, and DataSets available as open-source.

Reviewer comment (Novelty):
Would the authors consider expanding the related work section a bit including highly influential, but more non-lean related research?

Author response:
Yes, we're enhancing our related works section by including further discussions on papers related to theorem provers beyond Lean.

Reviewer comment (Evaluation):
What is the time used by LeanDojo and the optimized version vs. the sampling? Can this be added to Table 1

Author response:
In every case, we set a time limit of 10 minutes.

Reviewer comment (Writing):
The writing can still be improved. For example, the \cite or \citep should not be used interchangeably, and there should be a space before each inline citation. Also, some sentences can use some polishing, e.g., 'where anyone can put the formal statement in Lean for their mathematical theorem' on page 2.

Author response:
**W1**: Thanks for pointing this out we now have corrected the citations and also made other suggested changes in the paper.

Reviewer comment (Novelty):
Related prior work in tactic prediction in other systems (Coq, HOL4) should have been mentioned and compared. In particular, some considerations between atomic and compound tactics have been discussed in prior work in Coq (https://arxiv.org/abs/1905.09381, https://proverbot9001.ucsd.edu).

Author response:
**W2:** Thanks for the suggestion, we will be adding the related works of theorem provers other than Lean in our paper.

Reviewer comment (Experiments):
The claim of 'a new state-of-the-art performance of 30.6% on MiniF2F using Lean' is not entirely accurate, as the HyperTree Proof Search (HTPS) paper has already achieved over 40% success rate over the same dataset. I understand that your approach does not use reinforcement learning nor the same amount of computation resources as in the HTPS paper, but it might be better to make those assumptions clear and perhaps draw a more detailed comparison against HTPS.

Author response:
**W3:** Thank you for pointing this out. We acknowledge that our best performance is only for pass@1, as we did not conduct experiments for higher pass@k. We will correct our statement accordingly.

Reviewer comment (Evaluation):
Discussions: would it be possible to have a length distribution comparison between the generated proofs from Dynamic Sampling and Fixed Sampling models? Some qualitative examples to illustrate the differences between these two sampling methods would be highly appreciated.

Author response:
**Q2:** Yes, we have compared the lengths of proofs in Dynamic Sampling and Fixed Sampling methods, and we will be including these comparisons in the paper.  

**[Q2]**
We have included two plots in the discussion section that compare the tree structure and proof sizes between the dynamic and fixed sampling methods. Our findings from these plots indicate that the number of nodes at various depths of the tree is more widely distributed in dynamic sampling. This broader distribution allows for exploration of proofs of varying lengths, unlike the fixed sampling method which primarily focuses on shorter proofs.

Another plot, comparing the proof sizes of theorems proven using both methods, reveals that the ratio of proof sizes between dynamic and fixed sampling increases as the size of the proofs becomes longer. This observation suggests that dynamic sampling excels at exploring even longer proofs compared to fixed sampling.