Paper: ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving (ICLR-2024)

Reviewer comment (Presentation):
Adopting complicated mathematical fonts when unnecessary only takes from the readability of the paper. There is no need to use them when the default fonts suffice.

Author response:
We appreciate your feedback! We understand the reviewer's concern about the complexity of mathematical fonts making the paper less readable. Our intention was to maintain the mathematical accuracy and precision, e.g., our choice to use $\wp$ for "prompt" and $p$ for "program" was driven by a desire to avoid confusion, but we understand that this might have inadvertently complicated the presentation. To enhance readability, we will revise these symbols to more commonly used and easily comprehensible ones without losing the distinction between the two terms.

Reviewer comment (Experiments):
How should we improve upon ToRA? Since the pipeline relies on a proprietary model to generate the dataset, and the final performance of the strongest ToRA is still inferior to GPT-4, how can we further improve?

Author response:
Thank you for posing an insightful question about improving upon ToRA! One point that needs clarification is that the core method of ToRA is tool-integrated reasoning learning, and the data annotation in this process is **model-independent** for collecting tool-integrated reasoning trajectories. It can also be obtained through other open-source models or even manually annotated for higher-quality data.

Moreover, our experiments substantiated the effectiveness of this tool-integrated reasoning format on GPT-4, demonstrating its potential to further augment GPT-4's performance. As illustrated in Fig. 4 (right), the implementation of tool-integrated reasoning can yield a 9.8% improvement compared to PAL.

Looking forward, we see several main directions to further enhance mathematical reasoning based on ToRA:

- **Continue Pre-training**: We found in subsequent experiments that applying ToRA to models continue pre-trained on math and code data brings a more dramatic improvement! This suggests that pre-training and data engineering might deserve more attention for the research community!
- **SFT Mixture Scale-up**: We could explore amplifying the quality and coverage of fine-tuning data in tool-integrated reasoning format, which might further improve generalization and robustness.
- **Reward Modeling**: We could explore methods to enhance tool-integrated reasoning by incorporating reward modeling methods like [1], to further improve accuracy, intention alignment, interpretability, and to make it more user-friendly.

We extend our gratitude for your constructive feedback and the opportunity to clarify the nuances of our work. We believe that the proposed changes will significantly improve our paper's readability and impact.

[1] Lightman, Hunter, et al. "Let's Verify Step by Step." arXiv preprint arXiv:2305.20050 (2023).

Thank you for your continued interest in our research! We're excited to share our preliminary results from applying ToRA to models that are continue pre-trained (domain adaptive pre-training, DAPT [1]) on math and code corpus [2,3]. This has resulted in "ToRA-Code + DAPT”. As demonstrated in the table below, this approach has yielded an average improvement of 4.6% across all datasets, surpassing the previous state-of-the-art 7B model, ToRA-Code-7B.

Model         |Size| GSM8k| MATH| GSM-Hard | SVAMP | TabMWP | ASDiv | MAWPS | AVG
--------------|----|------|-----|------|-------|--------|-------|-------|-----  
ToRA-Code     |7B  | 72.6 | 44.6| 56.0 | 70.4  | 51.6   | 78.7  | 91.3  | 66.5
ToRA-Code + DAPT|7B  | **74.8** | **49.5** | **59.0** | **76.0**  | **63.5**   | **82.3**  | **92.8**  | **71.1**

We’ll will disclose more results in our future work. We sincerely value your review and guidance in improving our paper!

### References

[1] Gururangan, Suchin, et al. "Don't stop pretraining: Adapt language models to domains and tasks." *arXiv preprint arXiv:2004.10964* (2020).

[2] https://huggingface.co/datasets/open-web-math/open-web-math

[2] Azerbayev, Zhangir, et al. "Llemma: An open language model for mathematics." arXiv preprint arXiv:2310.10631 (2023).

Reviewer comment (Novelty):
**Limited of technical novelty**: Using imitation learning to improve the mathematical reasoning ability of open-source models has been proposed in many recent works, e.g.,  
- Scaling relationship on learning mathematical reasoning with large language models, https://arxiv.org/abs/2308.01825  
- WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct, https://arxiv.org/abs/2308.09583  
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models, https://arxiv.org/abs/2309.12284  
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning, https://arxiv.org/abs/2309.05653  
At the Output Space Shaping step, the authors use nucleus sampling to generate more reasoning paths and pick the corrected paths. This is a technique widely used in the works listed above. The only difference is that this paper fixes some of the preceding portions of wrong trajectories, while existing works resample the whole trajectory. However, validating which portions of trajectories are correct is very challenging, and the authors need to **enumerate** possible preceding portions of wrong trajectories, which is time-consuming. In my opinion, the existing method (i.e., re-sampling and picking the correct paths) is simpler and may be more effective.

Author response:
We appreciate your reference to concurrent works. While both ToRA and these studies employ imitation learning, the Tool-integrated Reasoning format proposed to train ToRA fundamentally differs from the CoT or PoT formats used in these works, yielding significantly better results.

Specifically, references [1-3] primarily adopt natural language rationale (CoT), focusing on augmenting CoT solutions [1] and questions [2, 3] for improvements. On the other hand, reference [4] concentrates on hybrid training using CoT and PoT.

In contrast, ToRA introduces the Tool-integrated reasoning format, which brings clear advantages. For instance, even the smallest ToRA-Code-7B outperforms MetaMath-70B [3] (44.6% vs. 26.0%) and MAmmoTH-Coder-34B [4] (44.6% vs. 43.6%).

We proposed the method of **learning from small model errors corrected by a teacher model**, which we call **teacher correction**, and we combined it with sampling for output space shaping. 

The step-level enumeration for teacher correction is a one-time cost for constructing training data. In our experiments, compared with using sampling alone for output space shaping, introducting teacher correction does not increase training cost because we maintain the same volume of training data.

Our research shows that integrating teacher correction contributes to higher efficacy than merely using sampling. As illustrated in Figure 5, employing the sampling strategy alone resulted in a 2.7% performance enhancement. Conversely, the incorporation of the correction strategy led to a performance improvement ranging from 3.4% to 4.0% across two datasets. This improvement was achieved while maintaining the same volume of training data as with sampling alone, thereby substantiating the effectiveness of our output space shaping strategies.

Reviewer comment (Reproducibility):
**Concerns on reproducibility**: As training data are not provided, reviewers and readers can't check the reproducibility of this paper. Note that existing works like WizardMath, MetaMath, and MAmmoTH have released their training data for the community to reproduce their results. Moreover, checkpoints of TORA are not provided in the appendix for checking reproducibility.

Author response:
We understand your concerns about reproducibility. We're currently undergoing an internal review to open-source the ToRA-Corpus and will add the open-source links to code, model checkpoints, etc., after the anonymous review period.

Reviewer comment (Experiments):
"TORA outperforms WizardMath by around 45% in Algebra and Number Theory, **which is attributed to stimulating and shaping tool-use behavior**." From Table 3, we cannot conclude that the better performance of TORA is due to stimulating and shaping tool-use behavior, as WizardMath uses augmented data **from LLaMA**, while TORA uses data generated **from GPT-4**. Note that GPT-4 is much more powerful than LLaMA.

Author response:
We appreciate your comments and would like to address the points raised regarding the performance of ToRA and WizardMath.

In the WizardMath paper, it is mentioned that they "for each instruction, … use ChatGPT and Wizard-E to generate 2~4 evolved instructions" and "depend on ChatGPT to provide process supervision" [2]. This suggests that WizardMath uses ChatGPT (not solely LLaMA) for data augmentation, as well as the human expert annotations within the original datasets (>96k in total). On the other hand, ToRA only uses GPT-4 annotated 16k data and applies output space shaping, and yet it achieves much higher performance (e.g., 44.6% vs. 10.7% on MATH). We would like to elaborate on why we attribute these gains predominantly to stimulating and shaping tool-use behavior.

- **Impact of tool-integrated reasoning in a fair setting**: Training LLaMA-2 7B on 16k annotations in Tool-Integrated Reasoning format (i.e., the ToRA-Corpus) is significantly better than using the 15k CoT annotations (33.6% vs. 7.2% in Fig 4), despite the fact that the CoT annotations come from human experts (i.e., the original annotations provided by the datasets) while ToRA Corpus is annotated by GPT-4 with an imperfect annotation accuracy of 83.1%. This demonstrates the effectiveness of tool-use behavior stimulation.
- **Additional augmented data used by WizardMath**: Even though WizardMath enhances human-annotated data with ChatGPT augmentation (>96k data in total), it still falls significantly behind using only the ToRA-Corpus (16k data in total) for training (33.6% vs. 10.7%, both base models are LLaMA-2 7B). We argue that this performance discrepancy isn't due to ChatGPT's inferiority to GPT-4, but rather the reasoning format.
- **Effect of shaping**: Additionally, our approach to shaping tool-use behavior has indeed resulted in substantial improvement. As Table 3 illustrates, this method boosted the performance of the 7B, 13B, and 34B models by 4.4%, 3.5%, and 3.4%, respectively on the competition-level MATH dataset. For instance, ToRA-34B rose from 47.4% to 50.8%. This enhancement was particularly notable in subtopics such as Precalculus, Geometry, and Algebra, with increases ranging from 5.9% to 6.3%.

Reviewer comment (Experiments):
In section 2.2, greedy decoding is used for generating trajectories from GPT-4. Thus, only one path per question can be obtained in TORA-CORPUS dataset. In my opinion, the accuracy of LLaMA trained on TORA-CORPUS has yet to saturate (e.g., plot the accuracy w.r.t. #samples of TORA-CORPUS). To generate more trajectories, a simple approach (which is widely used in the above works) is to use temperature sampling (rather than greedy decoding) and pick the correct ones for training.

Author response:
We appreciate your suggestion. In the original ToRA paper, we have already adopted the sampling method you mentioned to ensure more valid trajectories for each question.

**Kindly note**: In fact, right after the sentence you quoted, we stated, "For questions where GPT-4 fails with greedy decoding, we apply nucleus sampling with a sample size of 10 and keep up to four valid trajectories per question.”

Furthermore, Output Space Shaping is an algorithmic process that leverages sampling to introduce more diverse reasoning trajectories and introduce teacher correction to reduce reasoning errors. We believe the model used for sampling and the teacher for correction are not restricted to LLaMA models (despite that we used LLaMA in our experiments), but can also be GPT-4.

- We also note that there are follow-up works to ToRA that have replaced the teacher with GPT-4 and achieved good results [5].

Reviewer comment (Experiments):
As temperature sampling can generate more samples from GPT-4, TORA-CORPUS can be more diverse (compared with greedy decoding). To verify the effectiveness of Output Space Shaping in Section 3.5.2, it is better to **augment more data from GPT-4 and let the accuracy of LLaMA trained on the TORA-CORPUS saturates first**. Otherwise, it is difficult to say whether the improvement of LLaMA is from more training data or the proposed Output Space Shaping.

Author response:
We appreciate your suggestion. In the original ToRA paper, we have already adopted the sampling method you mentioned to ensure more valid trajectories for each question.

**Kindly note**: In fact, right after the sentence you quoted, we stated, "For questions where GPT-4 fails with greedy decoding, we apply nucleus sampling with a sample size of 10 and keep up to four valid trajectories per question.”

Furthermore, Output Space Shaping is an algorithmic process that leverages sampling to introduce more diverse reasoning trajectories and introduce teacher correction to reduce reasoning errors. We believe the model used for sampling and the teacher for correction are not restricted to LLaMA models (despite that we used LLaMA in our experiments), but can also be GPT-4.

- We also note that there are follow-up works to ToRA that have replaced the teacher with GPT-4 and achieved good results [5].

Reviewer comment (Experiments):
Ablation study of hyperparameter $n$ (maximum rounds). In experiments, $n=3$ is used. Question: is the performance of TORA sensitive to $n$?

Author response:
Our preliminary experiments found that on mathematical reasoning benchmarks like GSM8k and MATH, large language models nearly always solve problems with n ≤ 3 interactions, so we set it as the default hyperparameter. Setting a higher number of interactions does not bring significant gains.

Reviewer comment (Writing):
Writing issues:  
- "forward and backward reasoning, as well as result verification": references for forward/backward reasoning, verification  
- "Addressing these challenges requires complex symbolic reasoning over algebraic expressions": references for symbolic reasoning  
- where is the definition of $\theta$ in (4)?

Author response:
We appreciate your meticulous attention to details and pointing out writing typos. We will add all the references you mentioned and the definition of $\theta$ in equation (4) in future updates.

Reviewer comment (Novelty):
What is the difference between Tool-Integrated Reasoning (Algorithm 1) and existing methods (e.g., PAL (Gao et al., 2022), PoT (Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks, https://arxiv.org/abs/2211.12588))?

Author response:
As shown in Fig. 2, PAL and PoT are Program-based methods to solve tasks with program synthesis, while our proposed Tool-integrated Reasoning format integrates natural language reasoning with program-based tool use, in order to combine the benefits of both worlds.

As shown in Fig 4, the proposed Tool-Integrated Reasoning consistently surpasses Rationale-only and Program-only approaches. Remarkably, Using LLaMA-2, Tool-Integrated Reasoning achieves substantial improvements of 29.0% and 6.7% over Rationale-only and Program-only, respectively. With the closed-source GPT-4, the improvements are 19.1% and 9.8%, respectively. This emphasizes the effectiveness of integrating natural language rationales with programs.

Reviewer comment (Evaluation):
Are there any empirical results can support the ``Output Space Shaping improves **diversity**''. Furthermore, the diversity measure is not defined in the paper.

Author response:
Thanks for your question! We constructed many trajectories for each problem through sampling and teacher correction, where the reasoning processes are different but the answers are correct. In other words, improving diversity means increasing the number of different solution paths for the same problem.

Reviewer comment (Experiments):
From Figure 5, it can be observed that the performance of the model does not significantly decrease when output space shaping is removed. More experiments are needed to demonstrate whether the performance improvement in this stage is due to this training strategy rather than additional data and more training epochs.

Author response:
We appreciate your keen observation regarding Figure 5. There was a mistake in Figure 5 in the paper where the “sampling” strategy was Inappropriately referred to as “shaping”, and we will correct this in future updates. Here’s the clarification:

1. Output Space Shaping, in fact, brought a significant improvement. As shown in Table 3, this method improved the performance of the 7B, 13B, and 34B models by 4.4%, 3.5%, and 3.4% respectively. For instance, ToRA-34B increased from 47.4% to 50.8%. Furthermore, the improvement was particularly noticeable in subtopics like Precalculus, Geometry, and Algebra, with an increase of up to 5.9% ~ 6.3%.
2. Output space shaping includes two strategies: sampling and correction. In Figure 5, we conducted three experiments for comparison: pure imitation learning, using only the sampling strategy for shaping, and a combination of sampling + correction strategies for shaping. It's worth noting that all models were trained for the same number of epochs and both shaping experiments used the **same amount of training data**. The results showed that shaping using only sampling led to a 2.7% average improvement, while combining sampling + correction strategies led to an average improvement of 3.4% ~ 4.0% on two datasets under the same data amount. These results demonstrate the effectiveness of the sampling and correction strategies in our proposed output space shaping.

Reviewer comment (Evaluation):
It appears that the method proposed in this paper shows more significant improvements on smaller models, as the performance of the 7B, 13B, and 70B models shown in Figure 1 does not appear to differ significantly. How to explain this phenomenon?

Author response:
Thank you for your detailed observation! According to Table 2, ToRA-Code's performance on GSM8k increased from 72.6 to 75.8 and 80.7 as the model size increased from 7B to 13B and 34B, respectively. In contrast, the performance on the MATH dataset saturated as the model size increased, with performances of 44.6, 48.1, and 50.8, respectively.

To further understand the bottleneck encountered by ToRA on the MATH dataset at around 50%, we collected the accuracy of different models on MATH problems of different difficulty levels, as shown in the following table:

|  | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 |
| --- | --- | --- | --- | --- | --- |
| # Test Samples | 437 | 894 | 1,131 | 1,214 | 1,324 |
| Avg Question Length | 123.8 | 150.9 | 169.1 | 203.0 | 248.4 |
| ToRA-Code-7B | 77.3 | 62.3 | 50.0 | 40.9 | 22.3 |
| ToRA-Code-13B | 78.5 | 64.2 | 54.3 | 45.3 | 26.4 |
| ToRA-Code-34B | 82.4 | 66.9 | 59.5 | 46.8 | 27.3 |
| $\Delta$ 13B $\rightarrow$ 34B | +3.9 | +2.7 | +5.2 | +1.5 | +0.9 |
| GPT-4 PAL | 80.1 | 65.4 | 58.4 | 45.8 | 30.0 |
| GPT-4 Tool-integrated Reasoning | 89.5 | 77.7 | 71.0 | 55.6 | 39.0 |
| Training query coverage | 97.7 | 91.6 | 86.5 | 81.3 | 68.0 |

From the table, it is clear that the saturation of performance with increasing model size is primarily evident in difficult problems, i.e., levels 4 and 5. This phenomenon can be attributed to the following reasons:

1. Firstly, ToRA greatly improved the learning efficiency of smaller models, and raised the starting point of the scaling curve. As shown in Fig. 4, the format of Tool-integrated reasoning is more conducive to model learning under the same amount of training data, allowing smaller models to reach a very high level.
2. Secondly, the performance of the largest ToRA model is already close to that of the GPT-4 used for data annotation, and the end point of the scaling curve is mainly limited by the bottleneck of data annotation. The data constructed using GPT-4 has a relatively low coverage of difficult samples (81.3 and 68.0% for Level 4 and Level 5 samples, respectively). Increasing the size of the model increases the model's capacity, but the bottleneck is likely to be the quality and quantity of annotated samples.

This could encourage further research to focus on challenging math problems and more effective data engineering.

Reviewer comment (Novelty):
The alternating reasoning approach between natural language and tool usage proposed in this paper is fundamentally similar to the plan paradigm of Thought, Action, and Observation alternation in REACT [1]. Do you think this paradigm will become the dominant paradigm for agents to solve complex reasoning problems? [1] ReAct: Synergizing Reasoning and Acting in Language Models, ICLR 2023

Author response:
Thank you for your question! We agree with your view. Since language is the interface for LLMs to interact with tools and environments, the solving process for most LLM-Agent related tasks can be seen as interleaved rationale and interaction trajectory. Therefore, we believe alternating reasoning is a fundamental task completion paradigm based on the language interface, and we look forward to future work further promoting this method to solve more complex reasoning tasks with LLM!