Paper: Multilingual Mathematical Autoformalization (ICLR-2024)

Reviewer comment (Novelty):
The scope and novelty is limited.

Author response:
- As we specify in the Contributions subsection, the main contributions of this paper includes:
    1. Creating a large dataset
    2. Training multiple language models on it with different settings
    3. Evaluating these language models on held-out benchmarks manually to demonstrate that the dataset helps autoformalization 
    4. Conducting ablation experiments to show the multilinguality in formal languages benefits autoformalization.

  - We want to emphasise that this is a dataset paper, not a method one. The paper by Fu et. al proposes “model specialization”, which distils from a large model to a small model. Ours is about utilising the power of model distillation and back-translation to construct the most useful dataset for autoformalization, and validating various scientific points with it, as mentioned in the contributions.

Reviewer comment (Novelty):
This paper can be viewed as a back-translation version of Fu et al. (ICML 2023; see below for reference details).

Author response:
- We want to emphasise that this is a dataset paper, not a method one. The paper by Fu et. al proposes “model specialization”, which distils from a large model to a small model. Ours is about utilising the power of model distillation and back-translation to construct the most useful dataset for autoformalization, and validating various scientific points with it, as mentioned in the contributions.

  - Back-translation and sequence distillation are well-known techniques in the field of NLP and we do not make claims about inventing them. This is a dataset paper and we utilise these two techniques to create the dataset. The advantage of using back-translation for autoformalization is that informalising is much easier than formalising and hence we can get a high-quality dataset with it. Only the very first part of the process and experiments described in our paper is similar to Fu et. al and our novelty is elsewhere.

Reviewer comment (Experiments):
The proposed method is generic, but only evaluated on the narrow domain of autoformalization.

Author response:
- > The proposed method is generic, but only evaluated on the narrow domain of autoformalization.

     Autoformalization is a new domain that poses unique challenges like the lack of a sizeable dataset, the integration of neural and symbolic methods, and the difficulty of model evaluation. We think the domain is important enough to warrant its own treatment: Numerous papers solely on the specialised domain of autoformalization have been published, with lots in top conferences within both the machine learning and the theorem proving communities [1-8].

Reviewer comment (Novelty):
Three lines of related work are almost completely missing. Representative work in each line is listed below. Please conduct a literature search using the papers below as starting points and cite relevant papers in their references.

Author response:
- References: 
  - We note that autoformalization is quite different from both program synthesis and informal reasoning, with the aim of translating from a natural language to a formal one instead of creating formal programs from scratch. These papers are related to the method of creating the dataset, rather than the conclusions/contributions (which are about the dataset and the multilinguality) of the paper. We thank the reviewer again for the references. We have updated the paper to include these references and made a comparison of our paper to them. 

[1] Wang, Qingxiang, et al. "Exploration of neural machine translation in autoformalization of mathematics in Mizar." Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs. 2020.

[2] Wu, Y., Jiang, A.Q., Li, W., Rabe, M., Staats, C., Jamnik, M. and Szegedy, C., 2022. Autoformalization with large language models. Advances in Neural Information Processing Systems, 35, pp.32353-32368.

[3] Hahn, Christopher, et al. "Formal specifications from natural language." arXiv preprint arXiv:2206.01962 (2022).

[4] Reichel, Tom, et al. "Proof Repair Infrastructure for Supervised Models: Building a Large Proof Repair Dataset." 14th International Conference on Interactive Theorem Proving (ITP 2023). Schloss-Dagstuhl-Leibniz Zentrum für Informatik, 2023.

[5] Jiang, A.Q., Welleck, S., Zhou, J.P., Lacroix, T., Liu, J., Li, W., Jamnik, M., Lample, G. and Wu, Y., 2022, September. Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs. In The Eleventh International Conference on Learning Representations.

[6] Cosler, Matthias, et al. "nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models." arXiv preprint arXiv:2303.04864 (2023).

[7] Xin, Huajian, et al. "LEGO-Prover: Neural Theorem Proving with Growing Libraries." arXiv preprint arXiv:2310.00656 (2023).

[8] First, Emily, et al. "Baldur: whole-proof generation and repair with large language models." arXiv preprint arXiv:2303.04910 (2023).

Reviewer comment (Novelty):
**Model distillation** — Fu et al. Specializing Smaller Language Models towards Multi-Step Reasoning. ICML 2023.

Author response:
- We want to emphasise that this is a dataset paper, not a method one. The paper by Fu et. al proposes “model specialization”, which distils from a large model to a small model. Ours is about utilising the power of model distillation and back-translation to construct the most useful dataset for autoformalization, and validating various scientific points with it, as mentioned in the contributions.

  - Back-translation and sequence distillation are well-known techniques in the field of NLP and we do not make claims about inventing them. This is a dataset paper and we utilise these two techniques to create the dataset. The advantage of using back-translation for autoformalization is that informalising is much easier than formalising and hence we can get a high-quality dataset with it. Only the very first part of the process and experiments described in our paper is similar to Fu et. al and our novelty is elsewhere.

Reviewer comment (Novelty):
**Mathematical and logical reasoning with LLMs, and related work that involves multiple (natural) languages** — Cobbe et al. Training verifiers to solve math word problems. 2021 arXiv preprint: 2110.14168; Lewkowycz et al. Solving Quantitative Reasoning Problems with Language Models. NeurIPS 2022; Shi et al. Language Models Are Multilingual Chain-of-Thought Reasoners. ICLR 2023.

Author response:
- References: 
  - We note that autoformalization is quite different from both program synthesis and informal reasoning, with the aim of translating from a natural language to a formal one instead of creating formal programs from scratch. These papers are related to the method of creating the dataset, rather than the conclusions/contributions (which are about the dataset and the multilinguality) of the paper. We thank the reviewer again for the references. We have updated the paper to include these references and made a comparison of our paper to them. 

[1] Wang, Qingxiang, et al. "Exploration of neural machine translation in autoformalization of mathematics in Mizar." Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs. 2020.

[2] Wu, Y., Jiang, A.Q., Li, W., Rabe, M., Staats, C., Jamnik, M. and Szegedy, C., 2022. Autoformalization with large language models. Advances in Neural Information Processing Systems, 35, pp.32353-32368.

[3] Hahn, Christopher, et al. "Formal specifications from natural language." arXiv preprint arXiv:2206.01962 (2022).

[4] Reichel, Tom, et al. "Proof Repair Infrastructure for Supervised Models: Building a Large Proof Repair Dataset." 14th International Conference on Interactive Theorem Proving (ITP 2023). Schloss-Dagstuhl-Leibniz Zentrum für Informatik, 2023.

[5] Jiang, A.Q., Welleck, S., Zhou, J.P., Lacroix, T., Liu, J., Li, W., Jamnik, M., Lample, G. and Wu, Y., 2022, September. Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs. In The Eleventh International Conference on Learning Representations.

[6] Cosler, Matthias, et al. "nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models." arXiv preprint arXiv:2303.04864 (2023).

[7] Xin, Huajian, et al. "LEGO-Prover: Neural Theorem Proving with Growing Libraries." arXiv preprint arXiv:2310.00656 (2023).

[8] First, Emily, et al. "Baldur: whole-proof generation and repair with large language models." arXiv preprint arXiv:2303.04910 (2023).

Reviewer comment (Novelty):
**Executable program as formalisms of natural language** — In addition to OpenAI's Codex, the following work is also worth checking: Yu et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. EMNLP 2018; Austin et al. Program synthesis with large language models. 2021 arXiv preprint: 2108.07732; Fried et al. InCoder: A Generative Model for Code Infilling and Synthesis. ICLR 2023.

Author response:
- References: 
  - We note that autoformalization is quite different from both program synthesis and informal reasoning, with the aim of translating from a natural language to a formal one instead of creating formal programs from scratch. These papers are related to the method of creating the dataset, rather than the conclusions/contributions (which are about the dataset and the multilinguality) of the paper. We thank the reviewer again for the references. We have updated the paper to include these references and made a comparison of our paper to them. 

[1] Wang, Qingxiang, et al. "Exploration of neural machine translation in autoformalization of mathematics in Mizar." Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs. 2020.

[2] Wu, Y., Jiang, A.Q., Li, W., Rabe, M., Staats, C., Jamnik, M. and Szegedy, C., 2022. Autoformalization with large language models. Advances in Neural Information Processing Systems, 35, pp.32353-32368.

[3] Hahn, Christopher, et al. "Formal specifications from natural language." arXiv preprint arXiv:2206.01962 (2022).

[4] Reichel, Tom, et al. "Proof Repair Infrastructure for Supervised Models: Building a Large Proof Repair Dataset." 14th International Conference on Interactive Theorem Proving (ITP 2023). Schloss-Dagstuhl-Leibniz Zentrum für Informatik, 2023.

[5] Jiang, A.Q., Welleck, S., Zhou, J.P., Lacroix, T., Liu, J., Li, W., Jamnik, M., Lample, G. and Wu, Y., 2022, September. Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs. In The Eleventh International Conference on Learning Representations.

[6] Cosler, Matthias, et al. "nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models." arXiv preprint arXiv:2303.04864 (2023).

[7] Xin, Huajian, et al. "LEGO-Prover: Neural Theorem Proving with Growing Libraries." arXiv preprint arXiv:2310.00656 (2023).

[8] First, Emily, et al. "Baldur: whole-proof generation and repair with large language models." arXiv preprint arXiv:2303.04910 (2023).

Reviewer comment (Writing):
Term consistency needs double check: for example, both *formalisation* and *formalization* appear in the paper.

Author response:
- Spelling: 
  - We use British English spelling throughout, except for the word “autoformalization”, which has become a term.

Reviewer comment (Experiments):
Have you also checked potential data contamination issues? For example, would it be possible that GPT-4 has seen the test data in the evaluation benchmarks?

Author response:
- Data contamination:
  - This is a valid concern and we appreciate it. Three important pieces of information indicate that there is unlikely any data contamination with our paper:
    - We did not evaluate GPT-4. We only evaluated llama models.
    - The pre-trained model is not contaminated: We dedicate a subsection to the issue of data contamination with the pre-trained llama model in Section 6. Since it cannot even be instructed to write Isabelle or Lean, we think it unlikely that the pre-trained model is contaminated.
    - The fine-tuning datasets are not contaminated: The fine-tuning datasets come from Isabelle AFP + Lean Mathlib4. The test benchmarks come from miniF2F + ProofNet. These datasets are strictly disjoint. We use GPT-4 to create English descriptions of AFP and Mathlib4, which are extremely unlikely to include anything from the formal test benchmarks.
Since we have an unlikely contaminated pre-trained model, and an unlikely contaminated fine-tuning dataset, we conclude that the test evaluations are not affected by data contamination, and our conclusions stand.

Reviewer comment (Evaluation):
The evaluation could benefit from further enhancement. Currently, a sample size of only 50 examples from the benchmark is used, which may not provide sufficiently convincing results due to potential statistical limitations.

Author response:
- Evaluation: 
  - The main conclusions can be supported by 100 datapoints per model/inference configuration
    - We sample evaluation datapoints randomly (not cherry-picked)
    - With this much data, we already demonstrate very strong differences between the models to support our conclusions (16-18% vs 6-11% vs 0% of trivial corrections).
  - It’s incredibly expensive to annotate the entire benchmarks in such detail
    - We would like to point out that for this paper, we manually examine 50 examples from each benchmark x 2 benchmarks x 6 different model/language inference settings = 600 individual examinations. Evaluating each example requires 2-10 minutes for expert-level formal mathematicians, and in practice it took ~30 expert hours to finish. This is quite expensive. To execute the complete testing process, it requires 8-12x more effort, hence roughly 240-360 expert hours. This is USD 12K - 18K at a rate of $50/hr for formal mathematics experts (>= PhD level).
  - However, we do understand that a sample size of only 50 examples per benchmark (10% of the entire dataset) is smaller than traditionally done. Therefore to address the issue of low sample size, since we release the evaluation datapoints, we will open the evaluation up for the Isabelle and Lean communities. We have already identified Isabelle and Lean experts who are willing to contribute to this annotation endeavour. Leveraging the power of open-source communities is the best way to scale the evaluations while maintaining the quality of them.

Reviewer comment (Experiments):
As the author also mentioned, the synthesized datasets could be noisy. It would be beneficial to also include GPT4 in your comparative analysis for formalization quality. This would offer deeper insights into the noise levels within the generated datasets, i.e., how noisy it could be.

Author response:
- MMA dataset quality
  - Please see the overall comment for an in-depth analysis of the quality of the MMA dataset: https://openreview.net/forum?id=QqdloE1QH2&noteId=Z8PeCPad8b
  - To summarise the findings of the analysis:
    - Around 75% of the examined 200 problems were informalised completely correctly by GPT-4.
    - The failure cases are mostly due to ambiguity from the lack of appropriate context. And to ameliorate this issue is an exciting avenue for future research, possibly leveraging retrieval-augmented generation.

Reviewer comment (Experiments):
Considering that the total number of examples required for complete testing is approximately **eight times** the number of the sampled 50 examples used in the evaluation, I would appreciate more information about the overall effort required to execute the complete testing process.

Author response:
- Evaluation: 
  - The main conclusions can be supported by 100 datapoints per model/inference configuration
    - We sample evaluation datapoints randomly (not cherry-picked)
    - With this much data, we already demonstrate very strong differences between the models to support our conclusions (16-18% vs 6-11% vs 0% of trivial corrections).
  - It’s incredibly expensive to annotate the entire benchmarks in such detail
    - We would like to point out that for this paper, we manually examine 50 examples from each benchmark x 2 benchmarks x 6 different model/language inference settings = 600 individual examinations. Evaluating each example requires 2-10 minutes for expert-level formal mathematicians, and in practice it took ~30 expert hours to finish. This is quite expensive. To execute the complete testing process, it requires 8-12x more effort, hence roughly 240-360 expert hours. This is USD 12K - 18K at a rate of $50/hr for formal mathematics experts (>= PhD level).
  - However, we do understand that a sample size of only 50 examples per benchmark (10% of the entire dataset) is smaller than traditionally done. Therefore to address the issue of low sample size, since we release the evaluation datapoints, we will open the evaluation up for the Isabelle and Lean communities. We have already identified Isabelle and Lean experts who are willing to contribute to this annotation endeavour. Leveraging the power of open-source communities is the best way to scale the evaluations while maintaining the quality of them.