Paper: MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data (ICLR-2024)

Reviewer comment (Experiments):
The subset sampled for manual verification is small (only 88 data points). Given the variance in the dataset due to the different performance of the theorem prover on various types of math problems, the results on such a small dataset may not accurately represent the statistical characteristics of the whole dataset.

Author response:
**R2Q2. The subset sampled for manual verification is small.**

Thank you for the valuable comments. We expand the manual verification subset by randomly selecting and manually checking another 112 data points. 
The results of the 200 data points in total are demonstrated in Table 3 as follows.
It is shown that the results are consistent with more inspected data. 
(D4) and (D6) show significant differences in accuracy between the two groups.
Moreover, (D1) also shows significance with the inspected data scaled up.
The results demonstrate that the validated data have a higher quality than those without a formal validation process.

Table 3: Inspection dimensions and requirements in human evaluation. IS: Informal Statement. IP: Informal Proof. FS: Formal Statement. FP: Formal Proof. RT: Reasoning Type. 
Significant $p<0.005$ are marked with **bold**.

| Inspection Dimension | Requirement | Valid | Invalid | *p*-value | 
| ------- | ------- | ------- | ------- | ------- |
| (D1) IS Correctness | *Whether the informal statement is factually correct.* | 93.50 | 83.50 | **0.00167** |
| (D2) IS Relevance | *Whether the informal statement is relevant to each seed concept.* | 87.50 | 92.50 | 0.09604 |
| (D3) RT Classification | *Whether the informal statement is of the required question type.* |67.00 | 68.50 | 0.74903 |
| (D4) IP Correctness | *Whether the informal proof correctly solves the informal statement.* | 88.50 | 73.50 | **0.00012** |
| (D5) IS-FS Alignment | *Whether the informal statement and the formal statement describe the same problem and are aligned with each other.* | 74.00 | 66.50 | 0.10138 |
| (D6) IP-FP Alignment | *Whether the informal proof and the formal proof describe the same solution and have aligned proof steps.* |72.00 | 54.00 | **0.00018** |

Reviewer comment (Reproducibility):
It is unclear to me which model is used for generating the MUSTARD dataset. It would be helpful if the authors provided the performance of the LLM used for data generation as a reference.

Author response:
**R2Q3. The model used for generating the MUSTARD dataset.** 

We use GPT-4 [1] for generating the MUSTARD dataset. We have added the details to our revised version.

[1] OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023.

Reviewer comment (Experiments):
The scalability of the framework is still unknown. Only 7,335 data points are generated, which is a small token size for LLM fine-tuning, and the results after fine-tuning are mediocre. Furthermore, the effectiveness of this framework for larger LLM models (e.g., 34B or 70B models or GPT3 models) remains to be investigated.

Author response:
**R2Q4. The effectiveness of this framework for larger LLM models (e.g., 34B or 70B models or GPT3 models) remains to be investigated.** 

Thank you for the instructive comment. We further conduct experiments on Llama 2-70B. 
Since the Llama 2-34B and the GPT-3 parameters are not released, we are unable to experiment with these two models. 
The compared performances between Llama 2-7B and Llama 2-70B are demonstrated in Table 8 below. The results are also updated in Table 28 in Appendix F in the main paper. 
Due to time limitations, we demonstrate results on GSM8K first.
Llama 2-70B fine-tuned with MUSTARDSAUCE-valid performs 9.42\% higher than fine-tuned with MUSTARDSAUCE-random in the zero-shot manner, and 6.68\% higher in the few-shot manner. 
The performance gains are more significant compared to the results of Llama 2-7B.
Therefore, the proposed framework remains effective with larger fine-tuned language models. 
    
Table 8: Compared performances on GSM8K between Llama 2-7B and Llama 2-70B.

| MODEL            | Zero (G)                                       | Few (G)                                        |
| ---------------- | ---------------------------------------------- | ---------------------------------------------- |
| *Baselines*      |                                                |                                                |
| Llama 2-7B       | 7.2                                            | 12.8                                           |
| Llama 2-70B      | 31.7                                           | 54.1                                           |
| *Fine-tuning*    |                                                |                                                |
| Llama 2-7B > tt  | 9.0                                            | 15.5                                           |
| Llama 2-7B > in  | 8.3                                            | 14.4                                           |
| Llama 2-7B > ra  | 8.9                                            | 14.9                                           |
| Llama 2-7B > va  | 9.1 (+2.25%)  | 15.7 (+5.37%) |
| Llama 2-70B > tt | 36.6                                           | 55.8                                           |
| Llama 2-70B > in | 33.4                                           | 53.7                                           |
| Llama 2-70B > ra | 36.1                                           | 55.4                                           |
| Llama 2-70B > va | 39.5 (+9.42%) | 59.1 (+6.68%) |

Reviewer comment (Evaluation):
Since both natural language problem-solving and autoformalization are generated by LLM, and only ATP is used to verify the entire scheme, it is unclear how the authors handle inconsistencies between the final result calculated by the #reduce statement (as shown in Figure 2) and the natural language. The filtering strategy in Table 13 does not cover this situation (as ATP will not report an error).

Author response:
**R2Q7. The inconsistency between the #reduce statements and various theorem proofs.** 

Thank you for the question. 
The proportion of proofs with #reduce statements in the current MUSTARDSAUCE-valid set is 19.3%. 
MUSTARD aims to provide a uniform synthesis of theorem and proof data. 
Therefore, we have further filtered out the data points with the #reduce statements, which results in 5,922 data points.

Reviewer comment (Experiments):
There seems to be a significant misalignment between the generated 7,335 data points and the test set. For instance, the paper showcases a problem about the convergence of alternating series, which is within the scope of undergraduate mathematics, while the test set GSM8k only covers elementary school-level problems. Additionally, the paper mentions using the #reduce statement in Lean, which is an approach for math word problems but is not consistent with the style of various theorem proofs in Mathlib.

Author response:
**R2Q8. The test set GSM8K only covers elementary school-level problems.** 

Thank you for the valuable question. We have added experiments on the MATH dataset [2] that contain more challenging math word problems, and the results are demonstrated in Table 4 as shown above. 
Both GPT2-large and Llama 2-7B show significant improvements after fine-tuning with MUSTARDSAUCE-valid. For example, "GPT2-large > va" performs 40\% higher than "GPT2-large > ra" in the zero-shot manner. And "Llama 2-7B > va" has an 11.76\% increase compared to "Llama 2-7B > ra".
The observations further demonstrate the effectiveness of generated and validated MUSTARDSAUCE on math word problems.  

[2] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.

Reviewer comment (Reproducibility):
Which LLM is used for data generation?

Author response:
**R2Q3. The model used for generating the MUSTARD dataset.** 

We use GPT-4 [1] for generating the MUSTARD dataset. We have added the details to our revised version.

[1] OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023.

Reviewer comment (Experiments):
Does the method remain more effective when compared to fine-tuning the model using the entire generated dataset?

Author response:
**R2Q6. Does the method remain more effective when compared to fine-tuning the model using the entire generated dataset?** 

Thank you for the valuable comment. We have fine-tuned GPT2-large and Llama 2-7B with the entire generation dataset (contains 28,320 data points in total), and the results are demonstrated in Table 4 and Table 5 in the lines "GPT2-large > tt" and "Llama 2-7B > tt". 
We have also updated the results and marked them with blue in Tables 4 and 5 in the main paper.
We find that the models fine-tuned with the entire generated dataset perform better than those fine-tuned with MUSTARDSAUCE-random and MUSTARDSAUCE-invalid but are still inferior to those fine-tuned with MUSTARDSAUCE-valid. 
Therefore, more fine-tuning data improves the model performances as usual, but the smaller MUSTARDSAUCE-valid turns out to be more helpful. 
This finding again demonstrates our motivation that the validated theorem and proof data are of higher quality and are beneficial for reasoning.

Reviewer comment (Experiments):
Would the method potentially be more effective if more data points were generated and a larger model was employed for fine-tuning?

Author response:
**R2Q5. Would the method potentially be more effective if more data points were generated and a larger model was employed for fine-tuning?** 

Thank you for the constructive comment. 
Since generating and filtering new data with GPT-4 is costly, we instead use 75%, 50%, 25%, and 0% of current MUSTARDSAUCE-valid for fine-tuning Llama 2-7b to observe the effect of data scalability. 
We evaluate the model on all datasets. 
The results are demonstrated in Table 9 as follows. 
We have also uploaded the results in Figure 3 in the main paper. 
In general, the results on all datasets increase as the fine-tuning data scales up. 
Specifically, performances on the MUSTARD-test and mathlib show the most significant growth, and there is no decreasing trend in the growth rate.
Therefore we expect further performance improvements when more high-quality data are included. 


Table 9: Llama 2-7b performances with different data scales for fine-tuning. 

| Data Scale | GSM8K | MATH | mathlib | miniF2F | test |
| :--------- | :---- | :--- | :------ | :------ | :--- |
| 0%         | 12.8  | 2.6  | 0.0     | 0.0     | 0.0  |
| 25%        | 13.4  | 2.8  | 3.9     | 0.9     | 4.3  |
| 50%        | 14.2  | 3.1  | 5.3     | 1.4     | 7.5  |
| 75%        | 14.9  | 3.6  | 7.4     | 2.0     | 9.4  |
| 100%       | 15.7  | 3.8  | 8.7     | 2.9     | 12.2 |

Reviewer comment (Experiments):
It was also unclear to me which LLM is used for the informal data generation.

Author response:
We use GPT-4 [1] for generating both the informal and formal data.

[1] OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023.

Reviewer comment (Evaluation):
Could you elaborate on the low value of D3 in Table 3, the sentence on that in the paper was unclear to me?

Author response:
As introduced in Section 3.2, MUSTARD generates both theorem-proving problems and math word problems. D3 in Table 3 inspects whether MUSTARD accurately classifies the question types during generation. 
Specifically, we investigate instances where theorem-proving problems might be misclassified as math word problems, and vice versa, and assess if validation using Lean Prover reduces such misclassifications.
We find that the $p$-value of D3 is greater than 0.005, showing that Lean Prover validation does not significantly influence the classification.
Furthermore, the proportions in both Group Valid and Group Invalid in D3 are relatively lower than other inspection dimensions. One of the reasons is that MUSTARD generates a diverse range of problems from the elementary to higher educational levels, and generating theorem-proving problems at the elementary school level is specifically challenging.

Reviewer comment (Reproducibility):
Which LLM is used for the informal data generation?

Author response:
We use GPT-4 [1] for generating both the informal and formal data.

[1] OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023.

Reviewer comment (Theory):
If the LLM used is GPT-4 or another large off-the-shelf model, what would be the argument for this approach to help in performance of the best models? (if I misunderstood something, please correct me / give a short clarification)

Author response:
We sincerely appreciate your positive feedback! And we apologize for the misunderstanding of your question. This is a very interesting setup to see if the generated data by model M can benefit its own mathematical reasoning. Thank you for this valuable suggestion! We will surely add this experiment to our future study. For example, we will try this setup with Llama 2-70B and see how it works.