Paper: MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (ICLR-2024)

Reviewer comment (Novelty):
**Limited Technical Novelty**: The approach employed in developing MAmmoTH bears resemblance to previous works, such as Orca (Mukherjee et al., 2023). In situations where an adequate reservoir of mathematical Chain-of-Thought (CoT) or Program-of-Thought (PoT) data is unavailable, the model resorts to generating content from scratch. While this approach serves as a valuable resource and addresses the scarcity of specialized mathematics models, it may not be considered a groundbreaking contribution without a more rigorous evaluation and innovative techniques.

Author response:
A: Thank you for your question on the novelty of our work. First, we would like to highlight the fact that we did compare MAmmoTH directly with Orca-Platypus (a stronger variant of the original Orca)  in Table 3 and 4. As can be seen from the results, MAmmoTH significantly outperform it on both in-domain and OOD settings. This comparison is essential in establishing the superiority of MAmmoTH. The results clearly show that MAmmoTH outperforms Orca in handling a variety of mathematical reasoning problems, underscoring the advancements our dataset MathInstruct and model MAmmoTH bring to the field. 

Besides, we would like to highlight our contributions and novelty again:
- **Hybrid Rationale Integration:** A key innovation in MAmmoTH is the integration of both Chain-of-Thought (CoT) and Program-of-Thought (PoT) rationales, a feature not prevalent in models like Orca. This integration allows for flexible problem-solving strategies, enhancing computational precision and abstract reasoning capabilities in a single model​​.
- **Extensive and Diverse Dataset Curation:** MAmmoTH leverages the MathInstruct dataset, a comprehensive and diverse compilation of mathematical problems and rationales. This dataset covers a broader range of complexity levels and mathematical fields than typically seen, addressing a critical gap in existing math datasets and contributing to the uniqueness of our work​​.
- **Superior Performance on Comprehensive Benchmarks:** Our evaluations, as shown in Tables 3 and 4, demonstrate that MAmmoTH significantly outperforms advanced baselines like Orca-Platypus on both in-domain and out-of-domain datasets. This highlights MAmmoTH’s enhanced capabilities and generalization across various mathematical challenges​​.
- **Detailed Ablation Studies:** The ablation studies we conducted offer insights into the impact of different training subsets and model architectures, providing a deeper understanding of how various aspects of training data influence model performance. This level of analysis adds to the scientific rigor and novelty of our work​​.
We hope this response adequately addresses your concerns regarding the technical novelty of our work.

Reviewer comment (Experiments):
**Absence of an In-Depth Analysis of Training Data Distribution**: It is evident that each dataset contributes unevenly to the final training annotations. For instance, TheoremQA contains a mere 600 samples, raising questions about the true impact of incorporating such a small dataset. While it is reasonable to assume an enhancement in TheoremQA performance, it remains unclear whether this addition adversely affects the performance of other in-domain and out-of-domain tasks. Additionally, there is a concern that the model's focus on other mathematics datasets may hinder its proficiency in learning TheoremQA.

Author response:
A: We appreciate your constructive feedback and agree that a deeper analysis of training data distribution and influence is essential for a comprehensive understanding of our model’s performance. To respond, we added additional 9 model variants trained on specific subsets and combinations in Table 5:

|    Datasets                                        | GSM  | MATH| AQUA| NumG| SVA | Mat  	| Sim  	| SAT  	| MMLU| avg  	|
|------------------------------------------------|---------|--------|----------|---------|--------|----------|---------|--------|----------|---------|
| GSM                                            	| 56.6 	| 9.2  	| 24.4 	| 32.1 	| 65.4 	| 20.5 	| 12.3 	| 27.2 	| 25.2 	| 30.3 	|
| GSM + MATH                                   | 58.1 	| 28.2 	| 26.0 	| 34.7 	| 64.8 	| 50.1 	| 17.1 	| 28.6 	| 28.4 	| 37.3 	|
| GSM + MATH + TheoremQA           | 56.5 	| 26.5 	| 27.4 	| 35.5 	| 64.4 	| 50.6 	| 18.8 	| 29.1 	| 29.1 	| 37.5 	|
| GSM + MATH + Camel                    | 57.4 	| 28.5 	| 26.2 	| 37.5 	| 65.3 	| 50.4 	| 17.7 	| 29.3 	| 28.7 	| 37.9 	|
| GSM +   MATH + AQuA                             	| 56.1 	| 27.1 	| 37.8 	| 37.2 	| 64.8 	| 48.2 	| 19.8 	| 35.4 	| 39.8 	| 40.7 	|   	|
| GSM + MATH + Camel + AQuA                      	| 57.5 	| 29.1 	| 46.9 	| 42.2 	| 65.8 	| 49.6 	| 32.7 	| 42.3 	| 43.1 	| 45.5 	|
| MATH + Camel + AQuA + NumGLUE                  	| 24.7 	| 26.1 	| 39.4 	| 59.7 	| 61.6 	| 48.6 	| 43.4 	| 36.4 	| 41.2 	| 42.3 	|
| GSM + MATH + Camel + NumGLUE                   	| 50.8 	| 26.2 	| 20.9 	| 65.5 	| 65.8 	| 48.5 	| 41.4 	| 26.4 	| 24.7 	| 41.1 	|
| GSM + Camel + AQuA + NumGLUE                   	| 51.3 	| 14.8 	| 41.7 	| 58.8 	| 66.3 	| 31.0 	| 42.2 	| 34.1 	| 40.5 	| 42.3 	|
| GSM + MATH + Camel + AQuA + TheoremQA          	| 55.4 	| 28.6 	| 42.5 	| 44.9 	| 65.4 	| 50.8 	| 34.9 	| 41.3 	| 42.5 	| 45.1 	|
| GSM + MATH + Camel + AQuA + NumGLUE            	| 56.5 	| 28.9 	| 38.2 	| 63.7 	| 64.1 	| 47.9 	| 40.8 	| 38.6 	| 44.5 	| 47.0 	|
| GSM + MATH + Camel + AQuA + NumGLUE +TheoremQA 	| 53.8 	| 27.0 	| 38.2 	| 60.8 	| 65.9 	| 50.8 	| 41.8 	| 42.5 	| 42.7 	| 47.1 	|
| GSM + MATH + Camel + AQuA + NumGLUE + MathQA   	| 55.7 	| 28.8 	| 42.5 	| 62.1 	| 64.6 	| 45.9 	| 38.9 	| 41.3 	| 45.0 	| 47.2 	|
| MathInstruct                                           	| 53.6 	| 31.5 	| 44.5 	| 61.2 	| 67.7 	| 46.3 	| 41.2 	| 42.7 	| 42.6 	| 47.9 	|

Our key findings are summarized as follows:
- When the model is trained on individual datasets (GSM8K,  MATH, or AQuA), it demonstrates strong performance in tasks closely related to the respective training dataset. However, it generally struggles to generalize this performance to other tasks.
- The datasets GSM8K, MATH, and AQuA emerge as crucial components in the training process. Training the model on a combination of these datasets leads to a marked improvement in its overall performance, indicative of the complementary nature of these datasets. Omitting any one of these datasets results in a noticeable decline in overall performance.
- TheoremQA, despite its smaller scale, exerts some influence on the model's performance. It tends to enhance the model's capabilities on datasets like Mathematics and SimulEq, yet it may slightly impede performance on GSM8K and MATH. Nevertheless, its inclusion in the training regimen is favored, as it introduces a broader variety of questions and could potentially aid the model in generalizing to other complex problems that require advanced computational skills.
- Incorporating a wider range of datasets into the training consistently improves the model's performance across most tasks. This suggests that a broad and diverse knowledge base is instrumental in enhancing the model's ability to generalize.

We have included these findings in the revision.

Reviewer comment (Experiments):
**Insufficient Ablation Studies on Training Dataset Influence**: The majority of ablation studies appear to be centered around GSM8K, with the authors incrementally augmenting the training data size atop GSM8K. However, there is a notable absence of alternative strategies, such as initially training the model on MATH and subsequently introducing other datasets in a stepwise manner. Another informative ablation study could involve the removal of individual training datasets, such as M + C + A + N, and G + M + C + N, to elucidate the specific impact of each dataset on model performance.

Author response:
A: We appreciate your constructive feedback and agree that a deeper analysis of training data distribution and influence is essential for a comprehensive understanding of our model’s performance. To respond, we added additional 9 model variants trained on specific subsets and combinations in Table 5:

|    Datasets                                        | GSM  | MATH| AQUA| NumG| SVA | Mat  	| Sim  	| SAT  	| MMLU| avg  	|
|------------------------------------------------|---------|--------|----------|---------|--------|----------|---------|--------|----------|---------|
| GSM                                            	| 56.6 	| 9.2  	| 24.4 	| 32.1 	| 65.4 	| 20.5 	| 12.3 	| 27.2 	| 25.2 	| 30.3 	|
| GSM + MATH                                   | 58.1 	| 28.2 	| 26.0 	| 34.7 	| 64.8 	| 50.1 	| 17.1 	| 28.6 	| 28.4 	| 37.3 	|
| GSM + MATH + TheoremQA           | 56.5 	| 26.5 	| 27.4 	| 35.5 	| 64.4 	| 50.6 	| 18.8 	| 29.1 	| 29.1 	| 37.5 	|
| GSM + MATH + Camel                    | 57.4 	| 28.5 	| 26.2 	| 37.5 	| 65.3 	| 50.4 	| 17.7 	| 29.3 	| 28.7 	| 37.9 	|
| GSM +   MATH + AQuA                             	| 56.1 	| 27.1 	| 37.8 	| 37.2 	| 64.8 	| 48.2 	| 19.8 	| 35.4 	| 39.8 	| 40.7 	|   	|
| GSM + MATH + Camel + AQuA                      	| 57.5 	| 29.1 	| 46.9 	| 42.2 	| 65.8 	| 49.6 	| 32.7 	| 42.3 	| 43.1 	| 45.5 	|
| MATH + Camel + AQuA + NumGLUE                  	| 24.7 	| 26.1 	| 39.4 	| 59.7 	| 61.6 	| 48.6 	| 43.4 	| 36.4 	| 41.2 	| 42.3 	|
| GSM + MATH + Camel + NumGLUE                   	| 50.8 	| 26.2 	| 20.9 	| 65.5 	| 65.8 	| 48.5 	| 41.4 	| 26.4 	| 24.7 	| 41.1 	|
| GSM + Camel + AQuA + NumGLUE                   	| 51.3 	| 14.8 	| 41.7 	| 58.8 	| 66.3 	| 31.0 	| 42.2 	| 34.1 	| 40.5 	| 42.3 	|
| GSM + MATH + Camel + AQuA + TheoremQA          	| 55.4 	| 28.6 	| 42.5 	| 44.9 	| 65.4 	| 50.8 	| 34.9 	| 41.3 	| 42.5 	| 45.1 	|
| GSM + MATH + Camel + AQuA + NumGLUE            	| 56.5 	| 28.9 	| 38.2 	| 63.7 	| 64.1 	| 47.9 	| 40.8 	| 38.6 	| 44.5 	| 47.0 	|
| GSM + MATH + Camel + AQuA + NumGLUE +TheoremQA 	| 53.8 	| 27.0 	| 38.2 	| 60.8 	| 65.9 	| 50.8 	| 41.8 	| 42.5 	| 42.7 	| 47.1 	|
| GSM + MATH + Camel + AQuA + NumGLUE + MathQA   	| 55.7 	| 28.8 	| 42.5 	| 62.1 	| 64.6 	| 45.9 	| 38.9 	| 41.3 	| 45.0 	| 47.2 	|
| MathInstruct                                           	| 53.6 	| 31.5 	| 44.5 	| 61.2 	| 67.7 	| 46.3 	| 41.2 	| 42.7 	| 42.6 	| 47.9 	|

Our key findings are summarized as follows:
- When the model is trained on individual datasets (GSM8K,  MATH, or AQuA), it demonstrates strong performance in tasks closely related to the respective training dataset. However, it generally struggles to generalize this performance to other tasks.
- The datasets GSM8K, MATH, and AQuA emerge as crucial components in the training process. Training the model on a combination of these datasets leads to a marked improvement in its overall performance, indicative of the complementary nature of these datasets. Omitting any one of these datasets results in a noticeable decline in overall performance.
- TheoremQA, despite its smaller scale, exerts some influence on the model's performance. It tends to enhance the model's capabilities on datasets like Mathematics and SimulEq, yet it may slightly impede performance on GSM8K and MATH. Nevertheless, its inclusion in the training regimen is favored, as it introduces a broader variety of questions and could potentially aid the model in generalizing to other complex problems that require advanced computational skills.
- Incorporating a wider range of datasets into the training consistently improves the model's performance across most tasks. This suggests that a broad and diverse knowledge base is instrumental in enhancing the model's ability to generalize.

We have included these findings in the revision.

Reviewer comment (Evaluation):
**Lack of Comprehensive Error Analysis**: While MAmmoTH has demonstrated impressive performance, a detailed error analysis would provide valuable insights into areas where the model can further improve. The inclusion of illustrative examples could enhance our understanding of the model's strengths and limitations, aiding in the refinement of its capabilities.

Author response:
A: Thank you for your insightful feedback regarding the need for a comprehensive error analysis of the MAmmoTH model. We acknowledge the importance of this aspect and have taken steps to enhance our evaluation by including an error analysis. We meticulously categorizes the types of 40 errors encountered by MAmmoTH on GSM8K and MATH, providing a clearer understanding of its current limitations and areas for improvement. The errors are classified into several distinct categories:
Language Understanding Errors: These occur when the model fails to accurately comprehend the question or the context, leading to misinterpretation of the task at hand.
Grounding Errors: Such errors arise when the model incorrectly translates quantitative data or percentages (e.g., interpreting 150% as “2 times”).
Logic Errors: This category encompasses errors where the model's reasoning is flawed. 
Program Errors: These are errors where the model generate non-exectuable code due to the reasons like missing libraries or dependencies.
See the image in https://imgur.com/d1imghL for breakdown.

By classifying these errors, we aim to provide a clear picture of where MAmmoTH excels and where it requires further refinement. We also included specific examples of errors that MAmmoTH in the Appendix for better illustration. These examples are chosen to represent a range of issues.

Reviewer comment (Experiments):
How much of the training data would be filtered after the validation? It would be good to know the data utilization rate for the data generation process.

Author response:
A: We list the number of valid samples after filtering. AQuA-PoT (75%, 9772 out of 12881), MATH-PoT (51%, 7088 out of 14000), TheoremQA-PoT (44%, 703 out of 1600), TheoremQA-CoT (37%, 592 out of 1600), GSM8K (81%, 14591 out of 18000). We have included these statistics in the Appendix Table 6.

Reviewer comment (Evaluation):
I'm curious about whether the generated annotation converted from TheoremQA can be successfully executed, because the problems there would require many advanced calculations like integral and derivative computation.

Author response:
A: As we answered in the last question, around 44% of TheoremQA questions were successfully executed and matched the ground truth answer.

Reviewer comment (Experiments):
It is known that doing SFT on the dataset, especially with intermediate reasoning can do better. So even though MathInsutruct is already better than its competitors at generalizing, it can still remain a problem for even harder or unseen math problems.

Author response:
> Q: Even though MathInsutruct is already better than its competitors at generalizing, it can still remain a problem for even harder or unseen math problems.

A: We appreciate your comment on the limitations. As you rightly pointed out, and as we acknowledged in the Limitations section in the Appendix of our paper, despite the advancements brought by MathInstruct, the models might still encounter challenges with extremely complex or entirely unseen math problems. This is a common limitation when doing SFT. However, our study with MAmmoTH models demonstrates promising results in terms of performance on out-of-distribution (OOD) testing, suggesting potential pathways for enhancing model generalizability on unseen problems for future work:

**Broad Coverage and Diversity in Instruction Datasets:** When constructing instruction datasets like MathInstruct, ensuring diversity is crucial. This involves covering a broad range of mathematical problems from different fields and complexity levels. Such diversity helps in training models that are better equipped to handle a wide spectrum of mathematical challenges, enhancing their generalization capabilities.

**Hybrid Solution Types:** Incorporating a mix of different types of solutions, such as a combination of program-of-thought (PoT) and chain-of-thought (CoT) rationales, is more effective. This hybrid approach provides the models with varied perspectives and methods of problem-solving, which is particularly beneficial for tackling complex and novel mathematical problems.

Reviewer comment (Experiments):
Is there a baseline that just finetunes all of the 13 datasets into 1 model? I think this would give a better idea of how to access MathInstruct. If not that's fine.

Author response:
> Q: Is there a baseline that just finetunes all of the 13 datasets into 1 model? I think this would give a better idea of how to access MathInstruct. If not that's fine.

A: Before our study, there was no existing baseline model that had been fine-tuned on this specific collection of datasets. The MathInstruct dataset, as introduced in our paper, is an original contribution and was specifically compiled for this research.

MathInstruct is a meticulously curated dataset, comprised of 13 math datasets, including six that were newly curated by our team. This unique assembly of datasets is a novel contribution to the field of mathematical problem-solving with LLMs. The exceptional performance of our MAmmoTH models can be largely attributed to the high quality of MathInstruct. We believe that our approach of creating diverse, comprehensive datasets and incorporating a mix of solution types offers a promising direction for future research in this field.

Reviewer comment (Experiments):
The authors only experiment with one approach to hybrid prompting - it seems like a number of approaches could be viable, e.g. self-consistency across samples from both CoT+PoT, or letting the model pick which mode to operate in. If other approaches were tried but weren't effective, it would be good to see results (or at least a remark) indicating what was tried and justifying the chosen approach as the best one empirically.

Author response:
A: Thanks for bringing this up. We indeed have experimented with different prompting methods. For example, we tried “letting the model decide which prompting method (PoT vs. CoT) to use” and found that the performance was not quite satisfying. In contrast, our hybrid decoding (“first PoT, if fail, then CoT”) despite its simplicity leads to better performance in practice. We are happy to put more of our exploration results to justify our choice. 

Conceptually, our “hybrid decoding” is a special case of self-consistency decoding, where the ensemble size equals 2 (1 PoT + 1 CoT). We further tested other more expensive self-consistency variants like 10 PoT, 5 PoT + 5 CoT, 20 PoT, and 10 PoT + 10 CoT to see how far we could reach. We lay out some additional self-consistency experimental results on GSM and MATH for MAmmoTH-Coder-7B in the Appendix :

| 7B Coder Method                     | GSM  | MATH  |
|------------------------------------------|---------|-----------|
| 1 PoT                                        | 58.8   |   32.1   |
| 1 PoT + 1 CoT backup              |  59.4  |   33.4   |
| 5 PoT + 5 CoT majority vote     | 66.4   |  37.0    |
| 10 PoT majority vote                 | 67.0   |  37.8    |
| 10 PoT + 10 CoT majority vote | 69.6   |  38.2    |
| 20 PoT majority vote                 | 70.1   |  38.6    |

From our experiments, we show that self-consistency decoding can further improve performance. For the 7B MAmmoTH-coder, it seems that having a “PoT ensemble” already significantly improves the results to address the “non-executable issue“.

Reviewer comment (Experiments):
You mention that PoT is "activated" by using a particular prompt trigger. Does this mean the model won't generate PoT rationales without specifying this trigger, or will it generate a mix of both by default? (Insight on this question could address part of my second bullet in the Weaknesses section)

Author response:
A: That’s a very good catch! Since we appended PoT prompt triggers (e.g., “Let’s write a program.”) in the training questions, the model would be unlikely to generate PoT solutions without such triggers. We also tried removing such triggers and let the model automatically figure out the best type of rationale for different questions but we found such a planning task is actually very difficult for the model.

Reviewer comment (Presentation):
Connected to the question above, it would be nice to add an overall ratio of PoT/CoT to Table 1. Summing and dividing the listed sizes gives 27% PoT, which explain the need for the PoT trigger prompt, but I couldn't find this number in the paper - did I miss it somewhere?

Author response:
A: Thanks! We have updated the Table 1 to reflect the PoT/CoT ratio.

Reviewer comment (Reproducibility):
I actually train a 7B-CodeLLaMA with GSM8K0PoT training set prompted from GPT-3.5-turbo myself, the performance can definitely achieve 62% (>59.4 in Table 3). I'm not sure what would be the quality of this dataset. I don't mind sharing my GSM8k training set for the authors to reproduce.

Author response:
Thank you for your insightful observations and queries regarding the performance of CodeLlama - 7B on GSM8K. In response to your query, we have added additional experimental results in the Appendix Table 7:

|                                              | GSM8K | MATH | AQUA | NumG | SVA  | Mat  | Sim  | SAT  | MMLU | avg  |
|--------------------------------------|-------|------|------|------|------|------|------|------|------|------|
| CodeLlama - 7B - GSM8K         | **63.2**  | 18.4 | 25.4 | 32.9 | 67.6 | 40.1 | 16.3 | 26.4 | 26.1 | **35.2** |
| CodeLlama - 7B - MathInstruct  | 59.4  | 33.4 | 47.2 | 66.4 | 71.4 | 55.4 | 45.9 | 40.5 | 48.3 | **52.0** |

From the results, it is evident that training the CodeLlama 7B on the **individual GSM8K dataset can indeed yield better in-domain performance** compared with training on MathInstruct (**63.2 vs. 59.4**). This performance aligns with your observations from your own training experiments, and we appreciate you providing that as a baseline for comparison.

However, we also observe that training exclusively on a single dataset like GSM8K leads to challenges in generalizing to other datasets. The average performance of the CodeLlama - 7B model is significantly higher when trained on MathInstruct (52.0%) compared to being trained solely on GSM8K (35.2%). This demonstrates a substantial improvement in the model's generalization capability when trained on the diverse and comprehensive MathInstruct dataset. We report similar results in Table 5 based on the Llama-2 7B model.

These additional results provide strong evidence for the **quality and effectiveness of the MathInstruct dataset** in enhancing the performance of models across a range of mathematical reasoning tasks. While dataset-specific training can be effective for targeted tasks, the diverse and comprehensive nature of MathInstruct makes it a valuable resource for developing models with broad generalization capabilities in the domain of mathematical problem-solving.

Reviewer comment (Experiments):
In other words, I think the author should have more experiments to justify the datasets. For example, GSM8K PoT, we need to compare with a CodeLLaMA that trained on GSM8K PoT training set.

Author response:
Thank you for your insightful observations and queries regarding the performance of CodeLlama - 7B on GSM8K. In response to your query, we have added additional experimental results in the Appendix Table 7:

|                                              | GSM8K | MATH | AQUA | NumG | SVA  | Mat  | Sim  | SAT  | MMLU | avg  |
|--------------------------------------|-------|------|------|------|------|------|------|------|------|------|
| CodeLlama - 7B - GSM8K         | **63.2**  | 18.4 | 25.4 | 32.9 | 67.6 | 40.1 | 16.3 | 26.4 | 26.1 | **35.2** |
| CodeLlama - 7B - MathInstruct  | 59.4  | 33.4 | 47.2 | 66.4 | 71.4 | 55.4 | 45.9 | 40.5 | 48.3 | **52.0** |

From the results, it is evident that training the CodeLlama 7B on the **individual GSM8K dataset can indeed yield better in-domain performance** compared with training on MathInstruct (**63.2 vs. 59.4**). This performance aligns with your observations from your own training experiments, and we appreciate you providing that as a baseline for comparison.

However, we also observe that training exclusively on a single dataset like GSM8K leads to challenges in generalizing to other datasets. The average performance of the CodeLlama - 7B model is significantly higher when trained on MathInstruct (52.0%) compared to being trained solely on GSM8K (35.2%). This demonstrates a substantial improvement in the model's generalization capability when trained on the diverse and comprehensive MathInstruct dataset. We report similar results in Table 5 based on the Llama-2 7B model.

These additional results provide strong evidence for the **quality and effectiveness of the MathInstruct dataset** in enhancing the performance of models across a range of mathematical reasoning tasks. While dataset-specific training can be effective for targeted tasks, the diverse and comprehensive nature of MathInstruct makes it a valuable resource for developing models with broad generalization capabilities in the domain of mathematical problem-solving.

Reviewer comment (Experiments):
GSM8K has about 7k training data, how do you get 14k examples?

Author response:
Regarding your question about how we obtained 14k examples from GSM8K, which originally has about 7k training data: For each GSM8K question, we prompted GPT-4 to generate up to 3 PoT solutions. We further enhanced the dataset quality by meticulously removing near-duplicated solutions, ensuring a diverse range of unique problem-solving approaches. These GPT-4 synthesized programs were then rigorously vetted by comparing their executed results with human-annotated ground truth, thereby ensuring the high quality and reliability of the added rationales. We’ve added these clarifications in the revision.

We are grateful for your valuable feedback! Should there be any further queries or discussions needed, we are more than willing to engage and provide additional clarifications.