Paper: MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning (ICLR-2024)

Reviewer comment (Experiments):
I think it is really important to check whether or not the training dataset of GPT-4 contains datasets used in the paper. For instance, if GPT-4 is trained on one of the out-of-domain datasets considered in the paper, then it is very likely that GPT-4 copied the problems from the out-of-domain datasets and MathCoder used the copied problems for training. In this case, the results are not reliable because the baselines do not have access to the out-of-domain datasets.

Author response:
Thanks for your insightful comments. GPT-4 generated novel intermediate-level problems, and our training set and test set do not overlap; thus, the comparison with the baseline is fair.  Firstly, we evaluate the question through GPT-4 in Figure 4 in the paper, and 83.2% of interpolated problems are more challenging than GSM8K, and 95.6% of interpolated problems are simpler than MATH. Secondly, we directly check whether there is an intersection between the training set and the data of the five benchmarks. No two problems are exactly the same, so the new problems are not copied from the five datasets. 

To further address your question, we then use the Levenshtein distance [1,2] to detect whether there are similar questions. 
We sorted out the 5 most similar questions for each benchmark and found that even the most similar questions differed regarding decomposition ideas or calculation data [3]. Finally, it is worth mentioning that the baselines, WizardMath and RFT, also generate new questions using GPT-3.5 and GPT-4.

[1] Levenshtein, Vladimir I. "Binary codes capable of correcting deletions, insertions, and reversals." Soviet physics doklady. Vol. 10. No. 8. 1966.

[2] Chowdhery, Aakanksha, et al. "Palm: Scaling language modeling with pathways." arXiv preprint arXiv:2204.02311 (2022).

[3] https://anonymous.4open.science/r/Reviewer-Gqkc_Q1--CA9E/dedup.json

Reviewer comment (Theory):
There is no theoretical analysis or guarantee of the proposed method.

Author response:
Yes, the proposed MathCoder is an empirical study. Although it does not have theoretical support yet, we have empirically shown that the LCE block format can greatly improve the math reasoning accuracy on multiple public benchmarks (Table 2), and the proposed problem interpolation method can further improve math-solving models’ performance (Table 4, Table 8, and Table 9). Moreover, our method involves creating a diverse and high-quality math instruction dataset, followed by fine-tuning a large language model on this dataset. This approach is widely recognized as both common and effective [1,2].

[1]. Li, X., Yu, P., Zhou, C., Schick, T., Zettlemoyer, L., Levy, O., ... & Lewis, M. (2023). Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06259.

[2]. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., ... & Jiang, D. (2023). Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244.

Reviewer comment (Writing):
In Table 2, there is a typo as follows: Colsed-Source Model -> Closed-Source Model.

Author response:
Thanks for your warm reminder. We have revised the typo.

Reviewer comment (Novelty):
The proposed methods are comprised of multiple well-known effective components (coding helps, execution, consistency, distillation). Therefore, the contribution of this paper may not be significant enough.

Author response:
Thanks for your comments; we justify the difference with the examples you mentioned and highlight our contributions as follows:

**Relationship with the generation and execution of the Python code to solve math problems:** 
Yes, it is well known that existing work utilizing code execution to improve reasoning ability, such as PoT [1], PAL [2],  etc. However, those works are built on the closed-sourced models (e.g., GPT-3.5, GPT-4), while this paper focuses on improving the mathematical reasoning of open-source models (Llama2 and CodeLlama). Furthermore, typical code execution work mostly uses only one block of code (PoT [1], PAL [2]). In contrast, our method constructs the LCE format solutions, which interleaves the natural language and code chain for multiple times. This format can improve the mathematical reasoning of open-source models. We add the ablation study based on CodeLlama-7B, as shown in the table below. The LCE format solution results in a 17.9\% accuracy gain compared to code-only answers with program execution. In summary, no existing work presents the LCE-style instruction dataset for solving math problems. We add the ablation study in the revised version (Table 5, page 9) and present the results as follows:


| Data Format          | GSM8K | MATH | SVAMP | Mathematics | SimulEq | Average |
|:--------------------|:-----:|:----:|:-----:|:-----------:|:-------:|:-------:|
| LCE (ours)       | 67.8  | 30.2 | 70.7  |   55.8     | 49.6   | 54.8    |
| Only Code            | 50.2  | 20.2 | 61.6  | 39.8       | 12.8    | 36.9 **(-17.9)** |



[1]. Chen W, Ma X, Wang X, et al. Program of thought prompting: Disentangling computation from reasoning for numerical reasoning tasks[J]. Transactions on Machine Learning Research. (https://openreview.net/forum?id=YfZ4ZPt8zd)

[2]. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., ... & Neubig, G. (2023, July). Pal: Program-aided language models. In International Conference on Machine Learning (pp. 10764-10799). PMLR.


**Relationship with distillation of bigger models on GPT-4's solution:**

Distillation from bigger models (fine-tuned from GPT-4's solution) is not our main contribution; we use the distillation method to validate that the MathCoder models with LCE block can self-generate solution data without assistance from GPT-4. This is because GPT-4 is expensive. The MathCoder can maintain the `
`, `
` and `
` format and the solution from MathCoder can further improve mathematical reasoning capability of itself.

Reviewer comment (Novelty):
Using Python code to solve math problems is well known to have much more advantages over text prompts (Python code is more precise than text), as indicated by the paper "Program-aided Language Models" and many other papers.

Author response:
**Relationship with the generation and execution of the Python code to solve math problems:** 
Yes, it is well known that existing work utilizing code execution to improve reasoning ability, such as PoT [1], PAL [2],  etc. However, those works are built on the closed-sourced models (e.g., GPT-3.5, GPT-4), while this paper focuses on improving the mathematical reasoning of open-source models (Llama2 and CodeLlama). Furthermore, typical code execution work mostly uses only one block of code (PoT [1], PAL [2]). In contrast, our method constructs the LCE format solutions, which interleaves the natural language and code chain for multiple times. This format can improve the mathematical reasoning of open-source models. We add the ablation study based on CodeLlama-7B, as shown in the table below. The LCE format solution results in a 17.9\% accuracy gain compared to code-only answers with program execution. In summary, no existing work presents the LCE-style instruction dataset for solving math problems. We add the ablation study in the revised version (Table 5, page 9) and present the results as follows:


| Data Format          | GSM8K | MATH | SVAMP | Mathematics | SimulEq | Average |
|:--------------------|:-----:|:----:|:-----:|:-----------:|:-------:|:-------:|
| LCE (ours)       | 67.8  | 30.2 | 70.7  |   55.8     | 49.6   | 54.8    |
| Only Code            | 50.2  | 20.2 | 61.6  | 39.8       | 12.8    | 36.9 **(-17.9)** |



[1]. Chen W, Ma X, Wang X, et al. Program of thought prompting: Disentangling computation from reasoning for numerical reasoning tasks[J]. Transactions on Machine Learning Research. (https://openreview.net/forum?id=YfZ4ZPt8zd)

[2]. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., ... & Neubig, G. (2023, July). Pal: Program-aided language models. In International Conference on Machine Learning (pp. 10764-10799). PMLR.

Reviewer comment (Experiments):
Including execution of the Python code can improve performance, as it avoids LLM hallucinations on equations, similar to what is described in tool use or plugin papers.

Author response:
**Relationship with the generation and execution of the Python code to solve math problems:** 
Yes, it is well known that existing work utilizing code execution to improve reasoning ability, such as PoT [1], PAL [2],  etc. However, those works are built on the closed-sourced models (e.g., GPT-3.5, GPT-4), while this paper focuses on improving the mathematical reasoning of open-source models (Llama2 and CodeLlama). Furthermore, typical code execution work mostly uses only one block of code (PoT [1], PAL [2]). In contrast, our method constructs the LCE format solutions, which interleaves the natural language and code chain for multiple times. This format can improve the mathematical reasoning of open-source models. We add the ablation study based on CodeLlama-7B, as shown in the table below. The LCE format solution results in a 17.9\% accuracy gain compared to code-only answers with program execution. In summary, no existing work presents the LCE-style instruction dataset for solving math problems. We add the ablation study in the revised version (Table 5, page 9) and present the results as follows:


| Data Format          | GSM8K | MATH | SVAMP | Mathematics | SimulEq | Average |
|:--------------------|:-----:|:----:|:-----:|:-----------:|:-------:|:-------:|
| LCE (ours)       | 67.8  | 30.2 | 70.7  |   55.8     | 49.6   | 54.8    |
| Only Code            | 50.2  | 20.2 | 61.6  | 39.8       | 12.8    | 36.9 **(-17.9)** |



[1]. Chen W, Ma X, Wang X, et al. Program of thought prompting: Disentangling computation from reasoning for numerical reasoning tasks[J]. Transactions on Machine Learning Research. (https://openreview.net/forum?id=YfZ4ZPt8zd)

[2]. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., ... & Neubig, G. (2023, July). Pal: Program-aided language models. In International Conference on Machine Learning (pp. 10764-10799). PMLR.

Reviewer comment (Novelty):
Distillation of bigger models on GPT-4's solution is not a novel idea.

Author response:
**Relationship with distillation of bigger models on GPT-4's solution:**

Distillation from bigger models (fine-tuned from GPT-4's solution) is not our main contribution; we use the distillation method to validate that the MathCoder models with LCE block can self-generate solution data without assistance from GPT-4. This is because GPT-4 is expensive. The MathCoder can maintain the `
`, `
` and `
` format and the solution from MathCoder can further improve mathematical reasoning capability of itself.

Reviewer comment (Experiments):
One of the main contributions of the paper is the LCE solution format. However, even if we let GPT-4 generate solutions with demonstrations including reasoning, math, and code, it may not generate well-moduled solutions with clear blocked `
 
` and `
` sections. It is more likely to produce intersected solutions mixing the three together.

Author response:
We can guarantee that all samples in our train set have well-moduled solutions with clear blocked `
`, `
` and `
`. To maintain well-structured LCE solutions, we constructed the original datasets with the following steps:

Firstly, we prompted GPT-4 to solve the given math problem using code step by step and put the final answer with a specific prefix.

Secondly, we extracted the code in the response and executed it in our local environment. The texts after the extracted code were discarded to maintain the LCE format.

Thirdly, we concatenated the execution results to the prompt and model output and let GPT-4 continue to generate the problem-solving process.

We repeated the three processes until GPT-4 no longer outputs code and we detected the final answer. In this way, we can get a well-structured LCE solution.

Reviewer comment (Novelty):
How to maintain the well-structured LCE solution? If the method does not rely on the well-moduled LCE solution, then the method simply boils down to distillation on GPT-4's solution.

Author response:
Our model’s final performance heavily relies on the proposed LCE format solutions. We introduced special tokens `
`, `
` and `
` during training, and consequently, we can control the model to generate strict LCE format output during decoding.

Reviewer comment (Experiments):
What will the finetuning results be if the solution format is only Python coding, which is probably the major performance contributor?

Author response:
Thank you for your interest in our work. The average accuracy drops by 17.9 points when we use code-only format solutions to train the model. The results are presented in the table below. The experiments in this ablation study are based on CodeLlama 7B. This shows that the LCE format is an important performance contributor. We have updated the paper to include the results in Table 5 (page 9).

| Data Format          | GSM8K | MATH | SVAMP | Mathematics | SimulEq | Average |
|:--------------------|:-----:|:----:|:-----:|:-----------:|:-------:|:-------:|
| LCE (ours)       | 67.8  | 30.2 | 70.7  |   55.8     | 49.6   | 54.8    |
| Only Code            | 50.2  | 20.2 | 61.6  | 39.8       | 12.8    | 36.9 **(-17.9)** |

Reviewer comment (Evaluation):
MathCoder-Initial already has an accuracy of 77.3% on GSM8K and 44.0% on MATH. MathCoder is the model used to generate ground truth solutions for supervised finetuning smaller models. How do we explain why the performance on the distilled smaller models is better in Table 2?

Author response:
Thank you for your comments. The performance of the distilled smaller models is presented in the table below. As can be seen, their accuracy on GSM8K and MATH is all lower than MathCoder-Initial. We highlight that MathCoder-Initial is based on **CodeLlama-34B**.

| Model            | GSM8K | MATH |
| ---------------- | :-----: | :----: |
|**MathCoder-Initial** | **77.3** | **44.0** |
| MathCoder-L-7B   | 64.2  | 23.3 |
| MathCoder-L-13B  | 72.6  | 29.9 |
| MathCoder-CL-7B  | 67.8  | 30.2 |
| MathCoder-CL-13B | 74.1  | 35.9 |

Reviewer comment (Experiments):
During supervised finetuning, can you elaborate on how "cross-question masking" is implemented and how it helps performance compared with not using it?  
"In order to make the training more efficient, several instances are concatenated together to form a single input, while cross-question masking is used to ensure only tokens in the same instance are visible."

Author response:
The average token length of our 80k train set is 653 and 98.5% of the sample lengths are shorter than 2048. So, we set the token length as 2048. If we don’t concatenate different question-answer pairs together, each training sample would only contain one (question, solution) pair, and more than two-thirds of the tokens would be wasted on padding tokens. To address this issue, we concatenate several (question, solution) pairs into one sample to maximize the utilization of GPU[1][2]. Under this setting, the training can be sped up by 2.5x.

If we didn't modify the masking, the model could see prior questions and corresponding solutions when predicting the target solutions, similar to few-shot prompting, which would differ from the inference stage. So, tokens from previous samples are masked to eliminate distractions when predicting one question's solution. The masking is in the shape of:
```
1 0 0 0 0 0 0 0 0
1 1 0 0 0 0 0 0 0
0 0 1 0 0 0 0 0 0
0 0 1 1 0 0 0 0 0
0 0 1 1 1 0 0 0 0
0 0 1 1 1 1 0 0 0
0 0 0 0 0 0 1 0 0
0 0 0 0 0 0 1 1 0
0 0 0 0 0 0 1 1 1
```
In addition, the Llama models use Rotary Position Embedding, which means the attention scores are only related to the relative distance between two tokens. So, there is no gap between training and inference when concatenating several samples and using cross-question masking. 

The following table shows the influence of cross-question masking on performance. We adopted this technique mainly to improve training speed and efficiency. Cross-question masking is relatively straightforward, and we do not claim it as part of our contribution (see page 2 and page 3, Introduction).

|  Samples Data              | GSM8K | MATH  | SVAMP | Mathematics | SimulEq| Average        |
|:--------------------------|:-----:|:-----:|:-----:|:-----------:|:------:|:--------------:|
| w/ cross-question masking  | 67.8  | 30.2  | 70.7  | 55.8        | 49.6   |**54.8**        |
| w/o cross-question masking | 63.5  | 31.6  | 70.4  | 50.1        | 51.95  |**52.5** (-2.3) |

[1] Liu B, Chen C, Liao C, Gong Z, Wang H, Lei Z, Liang M, Chen D, Shen M, Zhou H, Yu H. MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning. arXiv preprint arXiv:2311.02303. 2023 Nov 4.

[2] https://github.com/bigcode-project/starcoder/blob/main/chat/train.py#L190

Reviewer comment (Experiments):
Another interesting experiment would be to show the improvement of the interpolation augmentation by measuring performance changes with different augmentation sizes. Do they always help? When will the improvement saturate?

Author response:
We thank the reviewer for raising this important point. Following your suggestion, we added experiments with different augmented sample sizes. The results are presented in the table below. As can be seen, there is a consistent improvement in average accuracy as the number of samples generated using problem interpolation increases. We have updated the paper to include this table (Table 9).

|  Samples Data        | GSM8K | MATH  | SVAMP | Mathematics | SimulEq| Average        |
|:--------------------|:-----:|:-----:|:-----:|:-----------:|:------:|:--------------:|
| 49k+0                | 50.6  | 22.9  | 53.2  | 46.0        | 29.6   |**40.5**        |
| 49k+11K              | 56.4  | 26.8  | 64.9  | 47.6        | 40.7   |**47.3** (+6.8) |
| 49k+31K (baseline)   | 67.8  | 30.2  | 70.7  | 55.8        | 49.6   |**54.8** (+14.3)|
| 49k+51K              | 68.0  | 32.6  | 70.9  | 60.1        | 52.7   |**56.9** (+16.4)|

Additionally, we believe there is space for further exploration. For example, we could use three or more different types of seed questions, further increasing the diversity of the generated questions. We leave this to future work.

Reviewer comment (Experiments):
To construct the training targets for D1 data, why using MathCoder-Initial instead of GPT-4? How is the performance if GPT-4 is used for generating the solutions? My hypothesis is that training on MathCoder-Initial's own predictions can amplify the model's own prediction mistakes, and it is helpful to explain more about why this can still improve the performance.

Author response:
**Q1**: To construct the training targets for D1 data, why using MathCoder-Initial instead of GPT-4? 

**A1**: Sorry for the confusion about our choice of MathCoder-Initial over GPT-4 for constructing the training targets of D1 data. 

Firstly, a very important reason for using self-distillation is to save the cost. Using GPT-4 is expensive, and we have spent over $7,000 on building the training set without calculating the cost for model training and testing. We have mentioned in the original paper that it "reduces the cost compared to using GPT-4" (see page 5, section 4.1). We would be grateful if you could understand the choice. 

Secondly, this choice can demonstrate the efficacy of the interpolation method in solving the problem as we can enhance the math-solving accuracy using data generated by our own model, though the models are less powerful than GPT-4.

Finally, using self-distillation further verifies that we have effectively trained our model to generate high-quality LCE solutions. This process further demonstrates our model's capability, showing that improvement through self-distillation is possible, which is also another major reason. 

To make the above reasons more sound, we have changed the original expression in page 4-5 of the revised paper. We hope this clears up your confusion on this issue.

**Q2**: How is the performance if GPT-4 is used for generating the solutions? 

**A2**: Following your suggestion, we subsequently used GPT-4 to generate the 31k data with some extra funding, creating a comparative dataset. We trained a version of CodeLlama-7/34B with this dataset, facilitating a direct comparison with our original model. The results are detailed in the table below. We have updated our paper to include the results in Table 10 (page 15).

| Base Model      | Data  | GSM8K| MATH | SVAMP | Mathematics | SimulEq | Average          |
|:---------------|:-----------------------|:----:|:----:|:-----:|:-----------:|:-------:|:----------------:|
| CodeLlama-7B    | 49k (GPT-4)+31k (MathCoder-Initial)         | 67.8 | 30.2 | 70.7  | 55.8        | 49.6    | 54.8             |
| CodeLlama-7B    | **80k (GPT-4)**                   | 68.4 | 31.2 | 76.3  | 61.6        | 52.5    | 58.0 **(+3.2)**  |
| CodeLlama-34B   | 49k (GPT-4)+31k (MathCoder-Initial)         | 81.7 | 45.2 | 82.5  | 75.9        | 65.8    | 70.2             |
| CodeLlama-34B   | **80k (GPT-4)**                   | 82.2 | 47.6 | 84.1  | 79.2        | 69.7    | 72.6 **(+2.4)**  |

As expected, using GPT-4 generated solutions leads to increased performance. With the 31k data generated from GPT-4 instead of MathCoder-Initial, our MathCoder- CL-7B and CL-34B's performances on all benchmarks can further increase. We have updated our paper with the results and the corresponding analysis in Appendix E.

**Q3**: My hypothesis is that training on MathCoder-Initial's own predictions can amplify the model's own prediction mistakes, and it is helpful to explain more about why this can still improve the performance.

**A3**: The experiments don't show the issue of error amplification by our MathCoder-Initial. One reason is that the MathCoder-Initial model is based on CodeLlama-34B, and it has already achieved an accuracy of 77.3% on GSM8K and 44.0% on MATH, demonstrating its inherent problem-generation ability. More importantly, to avoid amplification of the model's own prediction mistakes, we didn't take all the data generated by MathCoder-Initial. Instead, "we generate n different LCE solutions as depicted in (Wang et al., 2023a) for each new problem with our initial MathCoder models, keeping only those solutions for which all n answers match (n is set to 3 in this paper), thus ensuring our dataset's quality." We have revised the related paragraph to make it clearer.

Reviewer comment (Experiments):
The authors emphasize the importance of having natural language, code and execution results altogether in each problem solution. It would be good to have an ablation where the solutions only contain natural language or code, then we can check how much improvement is obtained by adding different components together in one solution.

Author response:
**Q4**: It would be good to have an ablation where the solutions only contain natural language or code; then, we can check how much improvement is obtained by adding different components together in one solution.

**A4**: We thank your suggestion for this meaningful ablation study. We list the results in the table below. Our new ablation study shows that the LCE format solutions outperform code-only format solutions by 17.9 points, and natural-language-only format solutions by 30.0 points. 

WizardMath 7B [1] serves as the performance of natural-language-only format solutions, as we do not have enough natural-language-only format data to train a new model. Results of WizardMath 7B are taken from [1]. We have updated our paper to include the results in Table 5 and Table 6 (page 9).

LCE format v.s. Code-only (Base model: CodeLlama 7B)

| Data Format          | GSM8K | MATH | SVAMP | Mathematics | SimulEq | Average |
|:--------------------|:-----:|:----:|:-----:|:-----------:|:-------:|:-------:|
| LCE (ours, 80k)       | 67.8  | 30.2 | 70.7  |   55.8     | 49.6   | 54.8    |
| Only Code (80k)           | 50.2  | 20.2 | 61.6  | 39.8       | 12.8    | 36.9 **(-17.9)** |


LCE format v.s. Natural-language-only (Base model: Llama 7B)
| Data Format          | GSM8K | MATH | SVAMP | Mathematics | SimulEq | Average |
|:--------------------|:-----:|:----:|:-----:|:-----------:|:-------:|:-------:|
| LCE (ours, 80k)       | 64.2  | 23.3 | 71.5  |  46.9     | 47.5   | 50.7    |
| WizardMath 7B [1] (96k, **NL**-format) | 54.9  | 10.7 | 36.1  | 9.3        | 12.8    | 24.8 **(-25.9)**|



[1] Luo, Haipeng, et al. "Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct." arXiv preprint arXiv:2308.09583 (2023).

Reviewer comment (Experiments):
To understand the importance of problem interpolation prompting, it would be helpful to have an ablation where the generated data is a mixture of data generated with GSM8K or MATH as the seed data, instead of using interporation.

Author response:
**Q5**: To understand the importance of problem interpolation prompting, it would be helpful to have an ablation where the generated data is a mixture of data generated with GSM8K or MATH as the seed data instead of using interpolation.

**A5**: Thank you for suggesting this interesting ablation study. To better understand the importance of the proposed problem interpolation via prompting, we have generated 31K data samples using GSM8K or MATH as the seed data with equal portions, without employing problem interpolation. Using this data, combined with the original D0 dataset, we trained the CodeLlama-7B and compared it with the result using problem interpolation. We present the performances in the table below.

| seed_data                           | GSM8K | MATH | SVAMP | Mathematics | SimulEq | Average |
|-----------------------------------|:-----:|:----:|:-----:|:-----------:|:-------:|:-------:|
| w/  interporation (MathCoder-CL-7B) | 67.8  | 30.2 | 70.7  | 55.8        | 49.6    | 54.8    |
| w/o interporation                   | 61.9  | 29.1 | 70.9  | 50.5        | 43.4    | 51.2 **(-3.6)** |



As shown above, the results with problem interpolation surpass those obtained without it in 4 out of 5 datasets, leading to a 3.6% accuracy improvement in average across the five datasets. This indicates that problem interpolation enhances accuracy of in-domain datasets such as GSM8K and MATH and aids out-of-domain datasets like Mathematics and SimulEq. Such generalization likely stems from the method's ability to bridge the gap between the more challenging MATH problems and the relatively simpler GSM8K problems.

We have updated our paper to include the results in Table 8 (page 14).

Reviewer comment (Experiments):
It is helpful to have an analysis on performance with different number of generated training samples, so that we can better understand how much improvement is possible with this approach.

Author response:
**Q6**: It is helpful to have an analysis of performance with different numbers of generated training samples so that we can better understand how much improvement is possible with this approach.

**A6**: Thank you for suggesting the analysis. We trained the CodeLlama 7B model with varying numbers of interpolated problem samples to answer the question. As shown by the results below, the average accuracy consistently improves with the increase in the number of problem interpolation data, which demonstrates the effectiveness of our method. We have updated our paper to include the results in Table 9 (page 14).

| Data Size              | GSM8K| MATH | SVAMP | Mathematics | SimulEq| Average |
|:------------------------|:------:|:------:|:-------:|:-------------:|:--------:|:---------:|
| 49k+0                | 50.6  | 22.9  | 53.2  | 46.0        | 29.6   |**40.5**        |
| 49k+11K              | 56.4  | 26.8  | 64.9  | 47.6        | 40.7   |**47.3** (+6.8) |
| 49k+31K   | 67.8  | 30.2  | 70.7  | 55.8        | 49.6   |**54.8** (+14.3)|
| 49k+51K              | 68.0  | 32.6  | 70.9  | 60.1        | 52.7   |**56.9** (+16.4)|