Paper: MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
**Potential for Benchmark Hacking:** Given the experimental setup, there is a slight risk that the proposed approach could lead to benchmark hacking.

Author response:
**A1.** As suggested, we perform a zero-shot evaluation on the DROP dataset to compare MetaMath with baseline models. Since we focus on mathematical reasoning, we only consider the DROP questions with numerical answers. The table below shows the testing accuracy. As can be seen, MetaMath-7B and MetaMath-13B still outperform the baseline models by a large margin, which shows MetaMath does not suffer benchmark hacking on GSM8K and MATH datasets. We have added the results in the updated paper (Table 10 in Appendix A.8).

| Method | #params | Accuracy (Exact Match) |
|  :---- |  :----: | :----: |
|SFT | 7B | 25.8 |
|RFT| 7B | 26.7 |
|WizardMath| 7B | 31.5|
|MetaMath|7B| **37.1**|
|||
|WizardMath|13B| 46.4|
| MetaMath |13B| **49.5** |

Reviewer comment (Experiments):
To some extent, both the weaknesses can be addressed by doing 0-shot evaluation on some other datasets like DROP (https://allenai.org/data/drop)

Author response:
**A1.** As suggested, we perform a zero-shot evaluation on the DROP dataset to compare MetaMath with baseline models. Since we focus on mathematical reasoning, we only consider the DROP questions with numerical answers. The table below shows the testing accuracy. As can be seen, MetaMath-7B and MetaMath-13B still outperform the baseline models by a large margin, which shows MetaMath does not suffer benchmark hacking on GSM8K and MATH datasets. We have added the results in the updated paper (Table 10 in Appendix A.8).

| Method | #params | Accuracy (Exact Match) |
|  :---- |  :----: | :----: |
|SFT | 7B | 25.8 |
|RFT| 7B | 26.7 |
|WizardMath| 7B | 31.5|
|MetaMath|7B| **37.1**|
|||
|WizardMath|13B| 46.4|
| MetaMath |13B| **49.5** |

Reviewer comment (Experiments):
In Table 3, MetaMath finetuning always begins with the AnsAug split, right? Do the authors have any thoughts on what would happen if we start training from (say) SV or FOBAR?

Author response:
**A2.** We apologize for the confusion caused by Table 3. When using data from multiple bootstrapping methods for finetuning, we mix all augmented data instead of using them sequentially (AnsAug -> Rephrasing -> SV -> FOBAR). In other words, at each training step, we randomly take a batch of samples from the mixed augmented data for updating the model parameters.

Reviewer comment (Experiments):
Some baseline approaches to compare are missing, e.g., [1, 2] and code-based LLMs like [3]

Author response:
**A1.** As suggested, we added the mentioned baselines in Table 2 of the updated paper and the additional comparisons are shown in the table below. As can be seen, MetaMath outperforms other CoT-based models and Code-LLaMA with the same number of parameters.

|     | \#params |  GSM8K | MATH |
|  :---- |  :----: |  :----: | :----: | 
| Code-LLaMA | 7B | 25.2 | 13.0  |
| MAmooTH-CoT | 7B | 50.5  | 10.4   | 
| MetaMath |7B|   66.5 | 19.8    |
|  ||    |      |
| Code-LLaMA |13B | 36.1 | 16.4 |
| Platypus | 13B |  25.7 |  2.5   | 
| MAmooTH-CoT | 13B|  56.3  |  12.9  | 
| MetaMath | 13B | 72.3  |  22.4  |
|  ||    |      |
| Platypus | 70B | 70.6 | 15.6  | 
| MAmooTH-CoT | 70B |   72.4  | 21.1  | 
| MetaMath | 70B | 82.3 |  26.6    |

Reviewer comment (Experiments):
The ablation study is not comprehensive enough. Only the 7B model is tested.

Author response:
**A2.** As suggested, we conducted an ablation study on LLaMA-2-13B and the results are shown below. The observation is consistent with our ablation study on the 7B model in Section 4.3: Combing answer augmentation and rephrasing augmentation data for finetuning leads to a slightly higher accuracy. The accuracy can be further improved by merging the FOBAR and SV augmentation data. We added the results in the updated paper (Table 9 in Appendix A.7).

| Method    | AnsAug | Rephrasing |  SV | FOBAR | GSM8K | MATH|
|  :---- |  :----: | :----: | :----: | :----: |   :----: | :----: | 
| SFT       |  &#10005; |  &#10005;  |  &#10005;  |  &#10005;   |50.9 |4.5  | 
| MetaMath  |  &#10003; |  &#10005;  |   &#10005;   |  &#10005; |66.0 |5.5  | 
| MetaMath  |  &#10005; |  &#10003;  |   &#10005;   |  &#10005; |67.5 |5.9  |
| MetaMath  |  &#10003; |  &#10003;  |   &#10005;   |  &#10005; |68.1 |5.8  | 
| MetaMath  |  &#10003; |  &#10003;  |   &#10003;   |  &#10003; |72.3 |7.2  |

Reviewer comment (Presentation):
Table 3 is confusing - should add a line breaker between SFT and MetaMath.

Author response:
**A3.** Thanks for your suggestion and we have fixed it accordingly in the updated version.

Reviewer comment (Novelty):
The major weakness I see is the lack of novelty. The paper in essence combines existing methods for bootsrapping.

Author response:
We would like to sincerely thank the reviewer for the useful comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

> Q1. **The major weakness I see is the lack of novelty. The paper in essence combines existing methods for bootsrapping.**

**A1.** This paper proposes a novel question bootstrapping idea to augment the training dataset. Compared with previous methods (e.g., SFT, RFT, WizardMath), our MetaMath introduces two novel augmentation methods:

(i) **Rephrasing questions.** Existing methods (e.g., RFT and WizardMath) focus on enlarging the answer data, which can be achieved by sampling more answers from LLMs. MetaMath proposes to create more questions, which is more challenging. To the best of our knowledge, **we are the first to use a rephrasing prompting method for augmenting questions**.

(ii) **Question bootstrapping by backward reasoning.**  Backward reasoning ability is crucial to solving many mathematical questions. However, the training set of mathematical tasks (e.g., GSM8K) lacks such data to improve backward reasoning. To deal with this problem, we introduce the templates (i.e., masking a number in the question and asking the LLM to predict the masked number) proposed in Self-Verification and FOBAR to create backward questions. **Note that both Self-Verification and FOBAR use backward reasoning for verification rather than data augmentation.** 

In summary, our paper proposes rephrasing questions and bootstrapping by backward reasoning to augment a diverse dataset --- MetaMathQA. MetaMath, which finetunes from state-of-the-art open-source LLMs on our MetaMathQA dataset, demonstrating excellent elementary mathematical problem-solving capability. In addition, we have released the MetaMathQA dataset for public use to improve the forward and backward reasoning capabilities of the LLM.

Reviewer comment (Experiments):
It is interesting that even reasoning paths with incorrect answers can be useful. Do you try to train using both correct and incorrect reasoning paths? Does this perform better than just correct?

Author response:
> Q2. **Do you try to train using both correct and incorrect reasoning paths? Does this perform better than just correct?**

**A2.** As suggested, we conducted an additional experiment on GSM8K to compare the performance of LLaMA-2 (7B) finetuned on (i) correct reasoning paths and (ii) correct+incorrect reasoning paths. The table below shows the testing accuracy. As can be seen, **finetuning on the correct paths is better than data containing incorrect reasoning paths.** 

|     | Accuracy| 
|  :---- |  :----:   | 
| Correct (20K)   |  56.2     | 
|Correct (20K) + Incorrect (7K) | 51.4 |

Reviewer comment (Experiments):
It is unclear how the proposed bootstrapping approach generalizes to other types of multi-hop reasoning problems.

Author response:
We would like to sincerely thank the reviewer for the valuable comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

> Q1: **It is unclear how the proposed bootstrapping approach generalizes to other types of multi-hop reasoning problems.**

**A1.** The core idea of the proposed bootstrapping approach is to diversify the questions in both forward and backward reasoning directions. Our approach can be extended to other reasoning tasks. We conducted an additional experiment to show a successful application of our bootstrapping method to the [Game of 24](https://en.wikipedia.org/wiki/24_(puzzle)), which involves multi-hop reasoning steps to reach 24 given 4 numbers. 
Given an original question with 4 numbers (2,3,4,12), its answer (2\*3 - 4) \* 12 is an expression that can reach 24.
We can apply answer augmentation and question bootstrapping to generate more question-answer pairs.

**Answer augmentation.** The solutions of obtaining 24 given 4 numbers may not be unique, e.g., 2\*12\*(4-3) = 24 is another solution to the given numbers (2,3,4,12). Hence, for a question with 4 numbers, we enumerate all the correct solutions and obtain the answer augmentation data.

**Question bootstrapping.** Game of 24 can be extended to **Game of $n$**, i.e., given 4 numbers (one number is 24), the goal is to obtain $n$ using basic arithmetic operations (+, -, \*, /). We use Game of n for question bootstrapping. We replace a number in the original question with 24 and the question is to obtain the number. This idea is similar to creating backward questions in our paper, i.e., masking a number in the question and asking the LLM to predict the number. For a Game of 24 question, we can bootstrap it and obtain 4 Game of $n$ questions, as an example shown in the table below.

|          | Bootstrapping1 | Bootstrapping2 | Bootstrapping3|Bootstrapping4| 
|  :----   | :----: | :----: | :----: | :----: | 
| Input (4 numbers)     | 24, 3, 4, 12   |  2, 24, 4, 12 | 2, 3, 24, 12  | 2, 3, 4, 24  |
| Target (n)  | 2 | 3 | 4 | 12|
| Solution (Valid Expression)      |     (4-3)/(12/24) = 2   | (24/12+4)/2 = 3| 24/12\*3-2 = 4 | (24/4-2)\*3 = 12 | 

**Game of 24 Setup.** We randomly select 1362 Game of 24 questions from 4num.com,
where 681 questions are for training and the remaining 681 questions are held-out for testing.
We apply the above augmentation methods to generate more training data from the 681 questions. We apply answer augmentation by enumerating all the correct forward solutions and obtain an AnsAug dataset consisting of 6052 question-answer pairs. We apply question bootstrapping to obtain a bootstrapping dataset (consisting of 2724 Game of n question-answer pairs). To verify the effectiveness of the bootstrapping approach, we randomly sample 4000 question-answer pairs (Game of 24) from the AnsAug datasets, and 2052 backward question-answer pairs (Game of n) from the bootstrapping dataset. We finetune LLaMA-2-7B on AnsAug and the mixed data separately for comparison.

**Results on Game of 24.** Table below shows the testing accuracy. As can be seen, our proposed augmentation approaches (AnsAug and AnsAug+Bootstrapping) have higher accuracy than SFT, which trains on the original 681 question-answer pairs. Furthermore, using question bootstrapping for augmentation can boost the performance of AnsAug. Hence, the proposed bootstrapping method is useful for Game of 24.

|            |  #Samples      | Accuracy  |
|  :----     |:----:     |  :----:   |
|SFT | 681 | 1.8 |
|AnsAug | 6052 | 10.2 |
|AnsAug + Bootstrapping | 6052 | **12.0** | 

**Results on Game of n.** For each question-answer pair in the testing set of **Game of 24**, we create 4 more testing questions of **Game of n** using the above question bootstrapping method. In total, we obtain 3405 testing questions. Table below shows the testing accuracy. Again, using our augmentation methods performs better than SFT by a large margin. Furthermore, AnsAug + Bootstrapping performs the best, demonstrating our proposed method is also useful for Game of n.

|            |  #Samples      | Accuracy  |
|  :----     |:----:     |  :----:   |
|SFT | 681 | 0.8 |
|AnsAug | 6052 | 3.0 |
|AnsAug + Bootstrapping | 6052 | **8.1** | 

We have included all the above experiments in Appendix A.4 of the updated paper.

Reviewer comment (Experiments):
The ablation of the method is not rigorously done. It is unclear if we keep increasing the number of AnsAug, we can get similar improvement.

Author response:
> Q2. **The ablation of the method is not rigorously done. It is unclear if we keep increasing the number of AnsAug, we can get similar improvement.I think it is necessary to show that increasing AnsAug to 395K cannot further increase the performance in order to prove the point made in the paper. I understand that this experiment can be costly, so doing this in a small scale to show the trend is good enough. I would love to see a curve on the accuracy vs. # of AnsAug and a curve on the accuracy vs # of a mixed of different augmentations.**


**A2.** Please note that the total number of MetaMathQA-GSM8K is 240K (as shown in Table 1). Therefore, we increase the AnsAug data to 240K (instead of 395K). We compare the performance of LLM finetuned on AnsAug data and MetaMathQA-GSM8K with question bootstrapping. The table below (and Figure 8 in Appendix A.6 of the updated paper) shows the trend on LLaMA-2-7B.
As can be seen, the trend is similar to Figure 2, i.e., finetuning on AnsAug rapidly reaches a state of accuracy saturation and continually increasing AnsAug data is difficult to boost performance further. In contrast, the test accuracy, when using bootstrapped questions, continues to exhibit a steady increase when AnsAug quickly saturates.

`LLaMA-2-7B`

| \# Samples     | 20K | 40K | 60K | 80K | 100K | 120K | 160K | 200K | 240K |
|  :----         |:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:| 
| AnsAug         |54.8  |58.1  |58.8  |59.2  |59.4  |60.2  |61.1  |60.7  |60.5  | 
| Bootstrapping   |56.2  |59.4  |61.1  |62.2  |63.0  |62.9  |65.5  |65.9  |65.8  |

In addition, we conducted additional experiments on LLaMA-2-13B and Mistral-7B [1] for further analysis. The experimental results are shown in following tables (also Figures 9 and 10 in Appendix A.6 of the updated paper). 
From the below tables, We can see that:

- For models with different scales and series, the proposed bootstrapping approach consistently surpasses the baseline method (AnsAug) by a large margin, showing question bootstrapping is effective in data augmentation.
- When the AnsAug data are abundant, bootstrapping is more effective. For instance, when there are only 20K AnsAug data for training, bootstrapping surpasses AnsAug by 1.4%, 1.7%, 0.8% for LLaMA-2-7B, LLaMA-2-13B and Mistral-7B, respectively. For the 160K AnsAug data case, Bootstrapping surpasses AnsAug by 4.4%, 4.8%, 6.4% for LLaMA-2-7B, LLaMA-2-13B and Mistral-7B, respectively.

`LLaMA-2-13B`
| \# Samples      | 20K | 40K | 80K | 120K | 160K |
|  :----          |:----:|:----:|:----:|:----:|:----:|
| AnsAug          |62.1  |64.8  |66.0  |66.6  | 67.0 |
| Bootstrapping   |63.8  |66.2  |68.5  |70.5  | 71.8 |


`Mistral-7B`
| \# Samples     | 20K | 40K | 80K | 120K | 160K |
|  :----         |:----:|:----:|:----:|:----:|:----:|
| AnsAug         |66.5  |67.1  |67.9  |68.9  |68.4  | 
| Bootstrapping   |67.3  |69.8  |71.2  |72.1  |74.8  |


[1] Jiang, Albert Q., et al. "Mistral 7B." arXiv preprint arXiv:2310.06825 (2023).

Reviewer comment (Experiments):
It is necessary to show that increasing AnsAug to 395K cannot further increase the performance in order to prove the point made in the paper. Although such an experiment can be costly, doing it on a small scale to show the trend is good enough.

Author response:
> Q2. **The ablation of the method is not rigorously done. It is unclear if we keep increasing the number of AnsAug, we can get similar improvement.I think it is necessary to show that increasing AnsAug to 395K cannot further increase the performance in order to prove the point made in the paper. I understand that this experiment can be costly, so doing this in a small scale to show the trend is good enough. I would love to see a curve on the accuracy vs. # of AnsAug and a curve on the accuracy vs # of a mixed of different augmentations.**


**A2.** Please note that the total number of MetaMathQA-GSM8K is 240K (as shown in Table 1). Therefore, we increase the AnsAug data to 240K (instead of 395K). We compare the performance of LLM finetuned on AnsAug data and MetaMathQA-GSM8K with question bootstrapping. The table below (and Figure 8 in Appendix A.6 of the updated paper) shows the trend on LLaMA-2-7B.
As can be seen, the trend is similar to Figure 2, i.e., finetuning on AnsAug rapidly reaches a state of accuracy saturation and continually increasing AnsAug data is difficult to boost performance further. In contrast, the test accuracy, when using bootstrapped questions, continues to exhibit a steady increase when AnsAug quickly saturates.

`LLaMA-2-7B`

| \# Samples     | 20K | 40K | 60K | 80K | 100K | 120K | 160K | 200K | 240K |
|  :----         |:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:| 
| AnsAug         |54.8  |58.1  |58.8  |59.2  |59.4  |60.2  |61.1  |60.7  |60.5  | 
| Bootstrapping   |56.2  |59.4  |61.1  |62.2  |63.0  |62.9  |65.5  |65.9  |65.8  |

In addition, we conducted additional experiments on LLaMA-2-13B and Mistral-7B [1] for further analysis. The experimental results are shown in following tables (also Figures 9 and 10 in Appendix A.6 of the updated paper). 
From the below tables, We can see that:

- For models with different scales and series, the proposed bootstrapping approach consistently surpasses the baseline method (AnsAug) by a large margin, showing question bootstrapping is effective in data augmentation.
- When the AnsAug data are abundant, bootstrapping is more effective. For instance, when there are only 20K AnsAug data for training, bootstrapping surpasses AnsAug by 1.4%, 1.7%, 0.8% for LLaMA-2-7B, LLaMA-2-13B and Mistral-7B, respectively. For the 160K AnsAug data case, Bootstrapping surpasses AnsAug by 4.4%, 4.8%, 6.4% for LLaMA-2-7B, LLaMA-2-13B and Mistral-7B, respectively.

`LLaMA-2-13B`
| \# Samples      | 20K | 40K | 80K | 120K | 160K |
|  :----          |:----:|:----:|:----:|:----:|:----:|
| AnsAug          |62.1  |64.8  |66.0  |66.6  | 67.0 |
| Bootstrapping   |63.8  |66.2  |68.5  |70.5  | 71.8 |


`Mistral-7B`
| \# Samples     | 20K | 40K | 80K | 120K | 160K |
|  :----         |:----:|:----:|:----:|:----:|:----:|
| AnsAug         |66.5  |67.1  |67.9  |68.9  |68.4  | 
| Bootstrapping   |67.3  |69.8  |71.2  |72.1  |74.8  |


[1] Jiang, Albert Q., et al. "Mistral 7B." arXiv preprint arXiv:2310.06825 (2023).

Reviewer comment (Experiments):
A curve on the accuracy vs. number of AnsAug and a curve on the accuracy vs. number of a mix of different augmentations would be desirable to observe performance trends.

Author response:
> Q2. **The ablation of the method is not rigorously done. It is unclear if we keep increasing the number of AnsAug, we can get similar improvement.I think it is necessary to show that increasing AnsAug to 395K cannot further increase the performance in order to prove the point made in the paper. I understand that this experiment can be costly, so doing this in a small scale to show the trend is good enough. I would love to see a curve on the accuracy vs. # of AnsAug and a curve on the accuracy vs # of a mixed of different augmentations.**


**A2.** Please note that the total number of MetaMathQA-GSM8K is 240K (as shown in Table 1). Therefore, we increase the AnsAug data to 240K (instead of 395K). We compare the performance of LLM finetuned on AnsAug data and MetaMathQA-GSM8K with question bootstrapping. The table below (and Figure 8 in Appendix A.6 of the updated paper) shows the trend on LLaMA-2-7B.
As can be seen, the trend is similar to Figure 2, i.e., finetuning on AnsAug rapidly reaches a state of accuracy saturation and continually increasing AnsAug data is difficult to boost performance further. In contrast, the test accuracy, when using bootstrapped questions, continues to exhibit a steady increase when AnsAug quickly saturates.

`LLaMA-2-7B`

| \# Samples     | 20K | 40K | 60K | 80K | 100K | 120K | 160K | 200K | 240K |
|  :----         |:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:| 
| AnsAug         |54.8  |58.1  |58.8  |59.2  |59.4  |60.2  |61.1  |60.7  |60.5  | 
| Bootstrapping   |56.2  |59.4  |61.1  |62.2  |63.0  |62.9  |65.5  |65.9  |65.8  |

In addition, we conducted additional experiments on LLaMA-2-13B and Mistral-7B [1] for further analysis. The experimental results are shown in following tables (also Figures 9 and 10 in Appendix A.6 of the updated paper). 
From the below tables, We can see that:

- For models with different scales and series, the proposed bootstrapping approach consistently surpasses the baseline method (AnsAug) by a large margin, showing question bootstrapping is effective in data augmentation.
- When the AnsAug data are abundant, bootstrapping is more effective. For instance, when there are only 20K AnsAug data for training, bootstrapping surpasses AnsAug by 1.4%, 1.7%, 0.8% for LLaMA-2-7B, LLaMA-2-13B and Mistral-7B, respectively. For the 160K AnsAug data case, Bootstrapping surpasses AnsAug by 4.4%, 4.8%, 6.4% for LLaMA-2-7B, LLaMA-2-13B and Mistral-7B, respectively.

`LLaMA-2-13B`
| \# Samples      | 20K | 40K | 80K | 120K | 160K |
|  :----          |:----:|:----:|:----:|:----:|:----:|
| AnsAug          |62.1  |64.8  |66.0  |66.6  | 67.0 |
| Bootstrapping   |63.8  |66.2  |68.5  |70.5  | 71.8 |


`Mistral-7B`
| \# Samples     | 20K | 40K | 80K | 120K | 160K |
|  :----         |:----:|:----:|:----:|:----:|:----:|
| AnsAug         |66.5  |67.1  |67.9  |68.9  |68.4  | 
| Bootstrapping   |67.3  |69.8  |71.2  |72.1  |74.8  |


[1] Jiang, Albert Q., et al. "Mistral 7B." arXiv preprint arXiv:2310.06825 (2023).