Paper: Magnushammer: A Transformer-Based Approach to Premise Selection (ICLR-2024)

Reviewer comment (Novelty):
Still, the main story of this paper is almost the same as LeanDojo.

Author response:
This is indeed a justifiable observation. However, note that our work was completed before LeanDojo’s release, and the two works should be treated as concurrent. Such similarities are often typical in scientific research when subsequent studies build upon existing work. ​​We showed how to effectively combine existing methods to achieve a generic premise selection tool that is a substantial improvement over Sledgehammer both in single- and multi-step settings (one-step proofs and Thor, respectively).
Also, please, see our answer to the question 1 of the reviewer XHfH where we say a bit about the differences between LeanDojo and our work.

Reviewer comment (Novelty):
The proposed two-step pipeline and training method are also commonly used.

Author response:
This is indeed a justifiable observation. However, note that our work was completed before LeanDojo’s release, and the two works should be treated as concurrent. Such similarities are often typical in scientific research when subsequent studies build upon existing work. ​​We showed how to effectively combine existing methods to achieve a generic premise selection tool that is a substantial improvement over Sledgehammer both in single- and multi-step settings (one-step proofs and Thor, respectively).
Also, please, see our answer to the question 1 of the reviewer XHfH where we say a bit about the differences between LeanDojo and our work.

Reviewer comment (Experiments):
It is great to know we could advance theorem proving by developing better premise selection approaches. My question is what the upper bound is given our current recipe. Is it possible to obtain the results of Thor with ground truth premise selection?

Author response:
Having an upper-bound would be indeed useful, however, there is no easy way to achieve it. The reason for this is that the proof steps generated with Thor often differ from the ground truth steps from the proof corpus, for which we do not have any “ground truth premises”.

Reviewer comment (Experiments):
Did you retrain Thor with Magnushammer or just apply the original Thor trained with Sledgehammer to Magnushammer?

Author response:
We retrain Thor with the original recipe. To clarify further, the training process of Thor does not involve Magnushammer at all: we simply use heuristics to find the steps in the proof corpus that were produced by Sledgehammer and replace them with an identifier sequence. How to improve on this recipe to make it Magnushammer-aware is an interesting research question that we leave for future work.

Reviewer comment (Evaluation):
Could you quantitatively evaluate the contribution of Magnushammer for theorem proving with Thor, such as how many proof steps call Magnushammer in your test?

Author response:
Thank you for this suggestion. Unfortunately, we have not logged such information. We will run our somewhat costly evaluations and provide numbers in the camera-ready version.

Reviewer comment (Experiments):
The main drawback seems to be the fact that the tools were evaluated only on two datasets (PISA and miniF2F).

Author response:
1. *What are other suitable datasets for the evaluation of tools for premise selection? The main drawback of the paper seems to be the fact that the tools were evaluated only on two datasets (PISA and miniF2F).*

In general, the premise selection problem appears both in the context of automated and interactive theorem proving. There are multiple provers in both categories, and they are typically based on different, often incompatible logics – therefore each of them is associated with different sets of benchmarks. For instance, one of the more prominent datasets for evaluating premise selection in automated theorem proving is this used in [1], which is derived from Mizar Mathematical Library, translated to TPTP [2] – the standard formalism used by automated theorem provers. On the other hand, ProofNet [3] is a recently released benchmark specific for the Lean interactive theorem prover that supports evaluation of several tasks, including automated proof search and premise selection.

In this work, we focused on premise selection for interactive theorem proving, and we decided to use the Isabelle’s ecosystem to develop and evaluate our approach. To the best of our knowledge, PISA and miniF2F benchmarks are the only published Isabelle’s datasets previously used to evaluate machine learning automation for Isabelle, thus we decided to evaluate our approach on them.

**References**

[1] Irving et al., DeepMath -- Deep Sequence Models for Premise Selection. NIPS 2016

[2] Sutcliffe: The TPTP World -- Infrastructure for Automated Reasoning. LPAR 2010

[3] Azerbayev et al., ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics. arXiv 2023

Reviewer comment (Experiments):
What are other suitable datasets for the evaluation of tools for premise selection?

Author response:
1. *What are other suitable datasets for the evaluation of tools for premise selection? The main drawback of the paper seems to be the fact that the tools were evaluated only on two datasets (PISA and miniF2F).*

In general, the premise selection problem appears both in the context of automated and interactive theorem proving. There are multiple provers in both categories, and they are typically based on different, often incompatible logics – therefore each of them is associated with different sets of benchmarks. For instance, one of the more prominent datasets for evaluating premise selection in automated theorem proving is this used in [1], which is derived from Mizar Mathematical Library, translated to TPTP [2] – the standard formalism used by automated theorem provers. On the other hand, ProofNet [3] is a recently released benchmark specific for the Lean interactive theorem prover that supports evaluation of several tasks, including automated proof search and premise selection.

In this work, we focused on premise selection for interactive theorem proving, and we decided to use the Isabelle’s ecosystem to develop and evaluate our approach. To the best of our knowledge, PISA and miniF2F benchmarks are the only published Isabelle’s datasets previously used to evaluate machine learning automation for Isabelle, thus we decided to evaluate our approach on them.

**References**

[1] Irving et al., DeepMath -- Deep Sequence Models for Premise Selection. NIPS 2016

[2] Sutcliffe: The TPTP World -- Infrastructure for Automated Reasoning. LPAR 2010

[3] Azerbayev et al., ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics. arXiv 2023

Reviewer comment (Writing):
The presentation of why the premise selection was considered difficult for LLMs, and how the proposed method tackles these difficulties is somehow weak. Some more examples and explanations/discussion on LLM/contrastive learning would help.

Author response:
6. *It would be nice if Section 6 provided more insightful characterization of the landscape of LLM-based theorem provers and how Magnushammer is positioned there. The explanation of the difficulty of premise selection by LLMs and how Mgnushammer overcomes them could be improved.*

We definitely agree that the field of “neural theorem proving” has become rich and diverse enough that it deserves more deep discussion, characterization, and comparative analysis. A special emphasis should be put on the premise selection problem, which is inherent to all the large corpora of formalized knowledge. The format of ICLR submission will not allow for a thorough treatment of the topic, but we are going to update the paper to better highlight the contributions of Magnushammer by contrasting it with different approaches. (Extending the paper turned out not entirely easy as the 9 pages are already filled compactly, but we are doing our best and will upload an updated version soon). Thank you, these were very meaningful suggestions!

Reviewer comment (Novelty):
Section 6 mentions GPT-f, PACT, Thor, and other LLM-based theorem-proving efforts, highlighting their diverse focuses. It would be nice if the authors provided a more insightful and coherent discussion on how different aspects are connected (e.g., premise selection, tactic prediction, proof step search), how the previous works explicitly or implicitly tackled the premise selection problem, and how the proposed method is advancing the area.

Author response:
As suggested by the reviewer, in the revised submission we extended and slightly reorganized Section 6 (Related work) in attempt to provide more coherent and insightful story surrounding the problem of premise selection, how other works tackle it, and how Magnushammer contrasts with them. We would be grateful for a feedback if the improved text sufficiently addresses the remarks of the reviewer. In order to comply with the space limitations, we removed the last short section (Conclusion) which mostly consisted of reiterating what was said elsewhere and brought less value than the extended picture of the related work. For the same reason, we also moved a paragraph concerning the ablation for re-ranking to Appendix C.5.

Reviewer comment (Experiments):
Section 3 Training part mentions that “This gives N-1+M negatives per proof state in one batch”. Is it possible that a positive premise for one state is also positive for another state in the same batch? Did you select positive examples intentionally avoiding this possibility, or, in this case, the negative examples are less than N-1+M?

Author response:
4. *Is it possible that a positive premise for one state is also positive for another state in the same batch? Did you select positive examples intentionally avoiding this possibility, or, in this case, the negative examples are less than N-1+M?*

That is a good question from the reviewer. In this case, such a premise is treated as a negative from the loss calculation point of view. However, it might be a false negative as in the scenario described by the reviewer. We didn't address this issue for simplicity and the fact that false negatives are omnipresent in the premise selection dataset. But we agree that it could be improved in our training pipeline. Nevertheless, despite this imperfect labeling, contrastive learning allows us to train models.

Reviewer comment (Experiments):
During the evaluation, for each proof state, Magnushammer parallelly checks whether a proof can be completed from premise subsets of different sizes. The author claims that similar techniques are also implemented in Sledgehammer. The author can explain more on this point to ensure the experimental comparison between Magnushammer and Sledgehammer is fair, e.g., in terms of parallelism, number of selected subsets, and their sizes.

Author response:
5. *Please, compare the strategies of trying different subsets of premises by Magnushammer vs. Sledgehammer.*

Magnushammer tries multiple subsets of premises as inputs to Isabelle’s tactics. In contrast, Sledgehammer inputs different subsets of premises as inputs to external provers, whereas the input of tactics consists of all the premises judged useful by the provers. The strategy of selecting the subsets of premises by Sledgehammer is complicated and not easily comparable to the strategy of Magnushammer. For instance, “the number of facts included in a problem varies from prover to prover, since some provers get overwhelmed more easily than others” [5].

Reviewer comment (Experiments):
The author mentioned that the MAPL dataset “decreases the probability of sampling false negatives while training contrastively”. Could the author provide further details on the likelihood of a premise being positive if it is not found by human-proof collections and Sledgehammer? Additionally, are there other methods to validate or enhance the confidence that it is a true negative? Also, the influence of false negatives on batch-contrastive learning can be elaborated.

Author response:
1. *The MAPL dataset “decreases the probability of sampling false negatives while training contrastively”. What is the likelihood of a premise being positive if it is not found by human-proof collections and Sledgehammer?*

This is an interesting and quite nuanced question. From the perspective of proof calculus, any premise may be seen as “positive", as it can be used in every proof, possibly in a “redundant” way. One could also argue that a premise is positive if it is contained in one of the minimal sets of premises appearing in the valid proofs. In practice, the automated methods often are unable to find a proof depending on such a minimal set of premises and some premises may not end up in a proof but still influence the automated search in a positive direction. Consequently, we can only think about an approximate notion of positive and negative premises, where positive means that including it increases the chances of automatically finding a proof, whereas negative means the opposite. Having said that, the number of new premises captured as positives (i.e., such that would previously be counted as negatives) on MAPL for theorems proved both by a human and Sledgehammer average to 9.85.

2. *Are there other methods to validate or enhance the confidence that a premise is a true negative?*

The only method (we can think of) of eliminating false negatives in premise selection is via repetitively running an automated prover on different sets of premises and collecting as many different proofs as possible. Doing this exhaustively is practically infeasible, so one strategy is to run the prover against sets of premises pre-selected by a data-driven / heuristical method, and by this exploring premises that are more likely to be positive.This strategy seems to be the best, nevertheless, it is still imperfect: some positive premises will not be captured by the pre-selection method, and therefore will constitute false positives.

3. *Please, comment on the influence of false negatives on batch-contrastive learning.*

In most practical scenarios (not only in premise selection) false negatives are unavoidable in batch-contrastive learning, especially in unsupervised settings. In practice, the presence of this issue does not prevent contrastive learning approaches from achieving good, often state-of-the-art, results on a range of different tasks [1,2]. Having said that, there are works that measure the negative influence of false negatives in contrastive learning [3,4]. This problem becomes especially pronounced when one applies hard negative mining in contrastive learning, i.e., sampling negatives that appear to be similar for the model. Hard negative mining is beneficial but also increases the probability of false negatives, which creates a trade-off (which [4] addresses).