Paper: Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification (ICLR-2024)

Reviewer comment (Novelty):
Most of the methods discussed in the paper are proposed by existing works and the main contribution of this paper is to try them out using the new OpenAI model. The contribution and novelty could be a weakness of the paper.

Author response:
Thanks for your comments. Our method draws inspiration from existing works. However, we think that there is novelty in our discussion, analysis, and method, which we hope you could consider. We will discuss this in four aspects:


First of all, we are the first to discover the important role code plays in GPT-4 Code Interpreter’s high performance in solving math problems via prompt engineering. GPT-4 Code Interpreter uses solutions interleaved with natural language, code, and execution results to solve math problems, possessing the ability to analyze execution results and self-repair. To the best of our knowledge, this is the first quantitative experimental analysis of this interesting phenomenon. 

Secondly, our method employs zero-shot prompting for automatic code-based self-verification, exploring GPT-4 Code Interpreter’s ability to use code multiple times, interleaved with natural language reasoning. Existing work on self-validation uses additional validator mechanisms ([3], [4], [5]), while our method employs zero-shot prompting for automatic self-verification. Additionally, our work uses code to verify the solution of math problems, which is different from existing work as well.

Thirdly, we employ weighted voting based on the results from the proposed code-based self-validation, thereby improving the performance of existing majority voting, as demonstrated in Figure 6. In contrast, most existing works use majority voting. This verification-guided weighted majority voting has never been studied before.

Fourthly, our paper focuses on the potential of using code multiple times, **interleaved with natural language reasoning**, while current methods using code, such as PAL [1] and PoT [2], generally only use code once. 

In summary,  although there are existing language models on code writing, self-validating, and majority voting, our method is notably different from theirs. We believe our approach of quantitative analysis and enhancing GPT4-Code with self-verification provides valuable and novel insights into the current top Large Language Models (LLMs). 

[1] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided language models. In International Conference on Machine Learning, pp. 10764–10799. PMLR, 2023.

[2] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, 2022.

[3] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.

[4] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023.

[5] Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification, 2023.

Reviewer comment (Novelty):
Most credit of the huge improvement on MATH should be given to the better capability of GPT4-code itself. The authors seem to over claim their own contribution throughout the paper. For example in Page 6, the authors said "Before the advent of GPT4-Code, prior frameworks (Lightman et al., 2023; Cobbe et al., 2021) depended on an external LLM to use natural language for verification and well-designed fewshot example prompts. In contrast, our approach simplifies the process by relying solely on a straightforward prompt for GPT4-Code, all in a **zero-shot** manner." But the root reason that zero-shot is applicable is because you are using OpenAI's instruction-tuned model. In addition, it is very likely that OpenAI has already used PRM (Lightman et al., 2023) to RL tune its new model, therefore the API you used could be based on top of Lightman et al., 2023.

Author response:
We are sorry that our paper has given you this impression. It is true that not all improvements are our contributions, which we have pointed out in our paper. We pointed out that GPT4-Code itself can achieve remarkable accuracy (69.69%) on the MATH dataset, as mentioned several times in the Introduction, Method, and Experiments sections. For example, in the first paragraph of Experiments (Section 4.1, Page 7), we note that GPT4-Code reaches 69.69% on MATH, largely surpassing the previous SOTA result (53.90%). This demonstrates that GPT4-Code exhibits strong abilities in solving math problems. We then used the GPT4-Code's result (69.69%) as **our baseline**.

To avoid misunderstanding and demonstrate our contributions more clearly, we have modified Tab. 1, Fig. 2, and Fig. 6 and adjusted expressions in many places in the paper. 

**In summary, our contributions are as follows**: We analyze the underlying code mechanism, develop a code-based self-verification (CSV) prompt, and introduce a weighted majority voting method. We also conduct quantitative experiments on various datasets (see detailed contributions in the Introduction). Based on your comments, we have more clearly acknowledged the high performance of the GPT-4 Code and analyzed it in depth. Additionally, we have consistently mentioned throughout the paper that our improvements are based on the GPT-4 Code.

Reviewer comment (Evaluation):
Some deeper qualitative analysis into what types of verification code are generated would also be helpful.

Author response:
Upon inspection of the generated solutions, we have found the following four types of verification code:

1. **Substitution**. The model substitutes the answer back to the question to check if it fits all the conditions. By inspecting the solutions we randomly sampled from the results, we found that this is the most common type of verification.

2. **Alternative Method**. The model uses a different method to solve the problem and check if it reaches the same final answer.

3. **Double Checking**. The model double-checks each part of the solution to see if they are correct.

4. **Approximation**. The model computes an approximation of the answer. This method is usually used when there is infinity involved in the problems. The model uses a large number to approximate the infinity to check if the result is close to the answer.

Corresponding examples are presented in an anonymous link [1]

[1] https://anonymous.4open.science/r/CSV_rebuttal_examples-7D47/verify_type_code.md

Reviewer comment (Experiments):
The paper considers both sampling and sequential self-repair. It would be great if the authors can analyze the tradeoff between the depths of self-repair (e.g., if self-verification continues to fail, how much can the model benefit from continuing self-repair until verification succeeds) versus breadths of self-verification (e.g., simply repair once, but generate multiple samples to do weighted sampling).

Author response:
Thank you for your constructive suggestions. Due to the limited window size of GPT4-Code, we cannot guarantee that the model will continue self-repairing until verification is successful. Consequently, we can only compare unlimited self-repair with majority voting of limited self-repair.

Majority voting with limited self-repair has a higher accuracy than unlimited self-repair. However, it also induces extra cost. To analyze the tradeoff between the depth of self-repair and the breadth of self-verification, we compare the results of two scenarios: one limiting the depth of self-repair to 1 and another allowing unlimited self-repair. The results are shown below:

|           | MATH Acc (%) |
|:----------:|:----------:|
| Unlimited (average depth 1.22) | 73.54 |
| self-repair once + voting (k=3) | 77.02 |

As shown above, when self-repair is limited to one attempt, voting reaches an accuracy of 77.02, which surpasses the accuracy of unlimited self-repair without voting by 3.48 percentage points. However, in the case of voting, the number of solutions generated for each problem is three. This is significantly higher than the average number of attempts in unlimited self-repair, which is 1.22.

Reviewer comment (Presentation):
The paper plots are visually misleading: Figure 2 accuracy numbers should all start from 0 as opposed to 60 in figure 2a and 40 in figure 2b. Otherwise the improvement looks like 10x as opposed to 6% comparing prompts 1 and 2. Similarly for other figures.

Author response:
We acknowledge your concern about the starting points of the y-axis in the accuracy graphs, which may unintentionally exaggerate the improvements. To minimize potential confusion, we have revised Figures 2a, 2b, and 6a to start from 0. We have also adjusted Table 1 by changing the baseline of voting to GPT4-Code + Voting. Further adjustments can be made if you have more suggestions.

Reviewer comment (Evaluation):
The paper lacks a qualitative analysis of the self-verification code and analysis of consistency between answer and verification code.

Author response:
Thank you for suggesting this interesting analysis. We did some qualitative analysis and provided examples of the consistency between the correctness of the solution process and the verification process. There are four cases:

The verification process is primarily consistent with the NL reasoning process. The model would start the verification by reevaluating the problem and providing a general plan for verification in natural language. Then, it would generate the code based on this plan. Finally, it would output the verification result after analyzing the output of the verification. The verification is a seamless continuation of the NL reasoning process.

To further analyze the consistency between the verification process, NL reasoning process, and output correctness, we have collected examples of four kinds listed in the table below. The reasoning process is correct if the answer before the verification is correct. The verification process is correct if the verification process itself is correct. We provide the examples with four anonymous links.  

|Case| Reasoning Process | Verification Process| Anonymous Link|
|:-------------------:|:-------------------:|:-----------:|:---------------|
| **1** | Correct | Correct     |  https://anonymous.4open.science/r/qualitative-analysis-of-consistency-between-verification-process-and-reasoning-process-0C64/examples/example1.md|
| **2** |  Correct   | Wrong      | https://anonymous.4open.science/r/qualitative-analysis-of-consistency-between-verification-process-and-reasoning-process-0C64/examples/example2.md|
| **3** |  Wrong   | Correct      | https://anonymous.4open.science/r/qualitative-analysis-of-consistency-between-verification-process-and-reasoning-process-0C64/examples/example3.md|
| **4** |  Wrong   | Wrong      | https://anonymous.4open.science/r/qualitative-analysis-of-consistency-between-verification-process-and-reasoning-process-0C64/examples/example4.md|

**1.**  When both the reasoning and verification processes are correct, the first verification successfully validates the solution, ensuring consistency with the answer and the natural language (NL) reasoning process. 

**2.**  If the reasoning is correct but the verification is erroneous, the first time verification fails to recognize the correct solution. 

**3.**  If the reasoning is incorrect but the verification is accurate, the verification process amends the erroneous reasoning, leading to a correct answer. 

**4.**  If both the reasoning and verification processes are incorrect, the verification is unable to rectify the flawed solution.  

Upon inspection of the generated solutions, we have found the following four types of verification code:

1. **Substitution**. The model substitutes the answer back to the question to check if it fits all the conditions. By inspecting the solutions we randomly sampled from the results, we found that this is the most common type of verification.

2. **Alternative Method**. The model uses a different method to solve the problem and check if it reaches the same final answer.

3. **Double Checking**. The model double-checks each part of the solution to see if they are correct.

4. **Approximation**. The model computes an approximation of the answer. This method is usually used when there is infinity involved in the problems. The model uses a large number to approximate the infinity to check if the result is close to the answer.

Corresponding examples are presented in an anonymous link [1]

[1] https://anonymous.4open.science/r/CSV_rebuttal_examples-7D47/verify_type_code.md

Reviewer comment (Novelty):
In terms of improving LLMs, the ideas of code writing, self-validating, and majority voting have been already explored in recent literature. The novelty of this work seems a simple combination of all three ideas together.

Author response:
**A1**: Thank you for your valuable feedback. We would like to highlight the novelty of our work from the following four aspects. 

First of all, we are the first to discover the important role code plays in GPT-4 Code Interpreter’s high performance in solving math problems via prompt engineering. GPT-4 Code Interpreter uses solutions interleaved with natural language, code, and execution results to solve math problems, possessing the ability to analyze execution results and self-repair. To the best of our knowledge, this is the first quantitative experimental analysis of this interesting phenomenon. 

Secondly, our method employs zero-shot prompting for automatic code-based self-verification, exploring GPT-4 Code Interpreter’s ability to use code multiple times, interleaved with natural language reasoning. Existing work on self-validation uses additional validator mechanisms ([3], [4], [5]), while our method employs zero-shot prompting for automatic self-verification. Additionally, our work uses code to verify the solution of math problems, which is different from existing work as well.

Thirdly, we employ weighted voting based on the results from the proposed code-based self-verification, thereby improving the performance of existing majority voting, as demonstrated in Figure 6. In contrast, most existing works use majority voting. This verification-guided weighted majority voting has never been studied before.

Fourthly, our paper focuses on the potential of using code multiple times, **interleaved with natural language reasoning**, while current methods using code, such as PAL [1] and PoT [2], generally only use code once. 

In summary,  although there are existing language models on code writing, self-validating, and majority voting, our method is notably different from theirs. We believe our approach of quantitative analysis and enhancing GPT4-Code with self-verification provides valuable and novel insights into the current top Large Language Models (LLMs).

Reviewer comment (Experiments):
Only one particular and proprietary LLM (i.e., GPT-4) is used for evaluation. Whether the results and findings reported in this work may generalize to other publicly available LLMs (e.g., Llama 2) or relatively smaller language models is unclear.

Author response:
**A3**: We have tested our method on CodeLlama 7B and 34B by applying the same prompting of CSV to these weaker models in a zero-shot manner. The results are shown in the table below. As can be seen, there are also noticeable improvements on both GSM8K and MATH, though the accuracy is much lower compared to GPT4-Code.

|                    | GSM8K Acc (%)|MATH Acc (%) |
|:------------------|:------------:|:-----------:|
| CodeLlama-7B       | 17.44        | 6.56        |
| CodeLlama-7B + CSV | 20.85        | 10.18         |
| CodeLlama-34B      | 28.96        | 9.12        |
| CodeLlama-34B + CSV| 37.6         | 13.36       |

Reviewer comment (Experiments):
Only one specific kind of benchmark (i.e., grade school math problems) is used, thus the proposed prompt might be overfitting to simple math problems. For the chosen benchmarks, improvements due to coding and CSV on GSM8K and MMLU-Math are already fairly small (i.e., 2-4%).

Author response:
**A2**： Thank you for your valuable feedback regarding the easier benchmarks. 

Regarding the concern that only one specific kind of benchmark (i.e., grade school math problems) is used, the MATH dataset consists of challenging competition-level problems of multiple math fields, including algebra, number theory, geometry, precalculus and so on [6]. Many of these problems are challenging even for college students, and the previous SOTA on this dataset is 53.9%, by the PHP method used on GPT4. GPT4-Code improves this score to 69.69%. Additionally, it is essential to note that the proposed prompt significantly improves the MATH dataset (69.69% -> 84.32%), demonstrating great potential for solving difficult math problems.

As for other relatively easier datasets, such as GSM8K and MMLU-Math, our proposed CSV + VW-Voting method demonstrates a smaller yet noticeable improvement compared to the MATH dataset. This relatively smaller improvement is not surprising, given these datasets' already quite high baseline performance. To fully appreciate the improvement induced by our method, we would like to calculate Relative error reduction **Ratio**, which is defined as $Ratio = \frac{Err_{baseline}-Err_{our}}{Err_{baseline}}$, where $Err = 1 - Acc$.

The results are demonstrated in the tables below. As can be seen, on GSM8K, our method reduces the Err by 4.1 percentage points, resulting in a Ratio of 57.7%. Experiments on MMLU-Math and MATH yield Ratios of 56.0% and 48.3%, respectively, showing a pattern similar to that observed with GSM8K.

| GSM8K                       | Err (%)| Ratio (\%) |
|:---------------------------|:------:|:---------:|
| GPT4-Code (baseline)        | 7.1    |  --       |
| GPT4-Code + CSV + VW-Voting | 3.0    | 57.7      |

| MMLU-Math                   | Err (%)| Ratio (\%) |
|:---------------------------|:------:|:---------:|
| GPT4-Code (baseline)        | 12.5   |  --       |
| GPT4-Code + CSV + VW-Voting | 5.5    | 56.0      |

| MATH                        | Err (%)| Ratio (\%) |
|:---------------------------|:------:|:---------:|
| GPT4-Code (baseline)        | 30.31  |  --       |
| GPT4-Code + CSV + VW-Voting | 15.68  | 48.3      |

Reviewer comment (Evaluation):
The improvement seems largely due to the superior capability of GPT-4. For instance, GPT-4 itself can outperform the state-of-the-art by a large margin without any sophisticated prompting and majority voting. It is already well-known that even a simple prompt could dramatically influence performance (see ). The particular finding of GPT-4 on the MATH benchmark is not very surprising.

Author response:
**A4**:  We agree that “even a simple prompt could dramatically influence performance.” However, how to perform “prompt engineering” to discover the appropriate prompt for making GPT-4 Code improve itself is not straightforward. We conducted a pioneering analysis to understand the key factor contributing to GPT-4 Code’s high performance on math problems and we propose the code-based self-verification prompt based on our analysis. In addition, we also found the new prompt is generalizable and can be used for CodeLlama to improve its math-solving accuracy as well. It is non-trivial to figure out such a generalized prompt. 

To make our contribution clearer, we have added the baseline of **GPT4-Code + Voting**, which is the result of naive majority voting and GPT4-Code without CSV. This baseline is 79.88%, which is shown in Table 1. All the methods that involve voting have been adjusted to be comparable to this baseline. We hope this will help readers better understand the contributions of this paper. We have listed the related results on  MATH below. Our method has a stable and noticeable effect based on the GPT4-Code baseline.

|                      | MATH Acc (%) | Improvement (%) |
|:--------------------|:------------:|:---------------:|
| GPT4-Code (baseline) | 69.69        | --              |
| GPT4-Code + CSV      | 73.54        | +3.85           |
 
|                                    | MATH Acc (%)| Improvement (%)|
|:----------------------------------|:-----------:|:--------------:|
| GPT4-Code + Voting (k=16, baseline)| 79.88       | --             |
| GPT4-Code + CSV + Voting (k=16)    | 83.54       | +3.66          | 
| GPT4-Code + CSV + VW-Voting (k=16) | 84.3        | +4.44          |

Reviewer comment (Presentation):
In Table 3, the last row only mentions _CSV_ (but not Voting), is it a typo? If not, why voting is not enabled?

Author response:
**A5**: Thank you for pointing this out. The absence of voting results was not a typo. Initially, due to our GPT-4 quota limits at the time of submission, we did not run MMLU-Math multiple times, which is necessary to obtain voting results. However, we have since conducted the experiment. We have updated Table 3 to include the outcome for GPT4-Code + CSV + VW-Voting on MMLU-Math, which achieves 94.5\% accuracy.

Reviewer comment (Experiments):
As-is the current paper has only one row in Table 1 that uses 16 samples with GPT4-Code, so the +14.63% improvement is a comparison between a 1 sample and 16 sample method. This can be a bit misleading to readers (or at least it was to me) since much of that improvement comes from using additional samples, which is something that could also be done through prior work via naive majority voting (which is actually evaluated later in the paper, in Fig 6b). Therefore, Table 1 should include a line for **GPT4-Code + CSV + Majority Voting** (which does about 1% worse than weighted voting, per Fig 6b).

Author response:
**A1**: Thank you for pointing this out. In Table 1, we have added the line for **GPT4-Code + CSV + Voting**, with an accuracy of 83.54%. We also added the line for **GPT4-Code + Voting**, with an accuracy of 79.88%. Here, Voting is an abbreviation for Naive Majority Voting. GPT4-Code + Voting improved the baseline by 10.19%. The fact that GPT4-Code + Voting can improve the baseline by 10.19% demonstrates that GPT4-Code generates diverse and accurate solutions. Additionally, the fact that GPT4-Code + CSV + Voting is higher than GPT4-Code + Voting by 3.66% is consistent with the improvement from GPT4-Code to GPT4-Code + CSV, which is 3.85%.

Also, we have noted that using the GPT4-Code result as the baseline might be misleading. Therefore, we have revised Table 1 to use GPT4-Code + Voting as the baseline for voting-related results instead.

|                     | MATH Acc (%)| Improvement (%)|
|:-------------------|:-----------:|:--------------:|
| GPT4-Code (baseline)| 69.69       | --             |
| GPT4-Code + CSV     | 73.54       | +3.85          |

|  | MATH Acc (%)                    | Improvement (%)|
|:----------------------------------|:--------------:|:-----:|
| GPT4-Code + Voting (k=16, baseline)| 79.88          | --    |
| GPT4-Code + CSV + Voting (k=16)    | 83.54          | +3.66 |
| GPT4-Code + CSV + VW-Voting (k=16) | 84.32           | +4.44 |

The tables above only list the accuracy across all the subjects. For the result of each individual subject, see Table 1 in the paper.

Reviewer comment (Experiments):
For the same reasons as in the previous comment, adding a line to Table 1 with **GPT4-Code + Majority Voting (no CSV)**, an ablation that is done with neither CSV nor weighted voting, would be an informative baseline to include. These two baselines would considerably strengthen the analysis.

Author response:
**A1**: Thank you for pointing this out. In Table 1, we have added the line for **GPT4-Code + CSV + Voting**, with an accuracy of 83.54%. We also added the line for **GPT4-Code + Voting**, with an accuracy of 79.88%. Here, Voting is an abbreviation for Naive Majority Voting. GPT4-Code + Voting improved the baseline by 10.19%. The fact that GPT4-Code + Voting can improve the baseline by 10.19% demonstrates that GPT4-Code generates diverse and accurate solutions. Additionally, the fact that GPT4-Code + CSV + Voting is higher than GPT4-Code + Voting by 3.66% is consistent with the improvement from GPT4-Code to GPT4-Code + CSV, which is 3.85%.

Also, we have noted that using the GPT4-Code result as the baseline might be misleading. Therefore, we have revised Table 1 to use GPT4-Code + Voting as the baseline for voting-related results instead.

|                     | MATH Acc (%)| Improvement (%)|
|:-------------------|:-----------:|:--------------:|
| GPT4-Code (baseline)| 69.69       | --             |
| GPT4-Code + CSV     | 73.54       | +3.85          |

|  | MATH Acc (%)                    | Improvement (%)|
|:----------------------------------|:--------------:|:-----:|
| GPT4-Code + Voting (k=16, baseline)| 79.88          | --    |
| GPT4-Code + CSV + Voting (k=16)    | 83.54          | +3.66 |
| GPT4-Code + CSV + VW-Voting (k=16) | 84.32           | +4.44 |

The tables above only list the accuracy across all the subjects. For the result of each individual subject, see Table 1 in the paper.

Reviewer comment (Experiments):
There doesn't seem to be a **GPT4-Code + CSV** row in Table 2, nor a **GPT4-Code + CSV + Voting** row in Table 3. This use of different setups on different datasets is confusing – an explanation should be given, or preferably both should be added. Having both of these novel methods from the paper evaluated on all 3 datasets would strengthen the paper.

Author response:
**A2**: Thank you for pointing this out. Due to our GPT-4 quota limits at the time of submission, we did not initially run MMLU-Math multiple times. However, we have since conducted the suggested experiment. The omission of the line for **GPT4-Code + CSV** on GSM8K was an oversight, which has now been corrected with the inclusion of the corresponding result, 94.5. Also, in Table 3, we have included the result for **GPT4-Code + CSV + Voting** on MMLU-Math, which is 94.5.

Reviewer comment (Presentation):
Figure 6a is quite confusing to me:  
- Which dataset is used? I believe it's MATH overall dataset, but this should be in the caption and optionally also main text  
- I don't understand why there are 5 separate paths graphed, given that there's nothing special differentiating path 0 from path 1, etc. The different path indices are just IID samples, so of course when you calculate precision/accuracy/recall averaged over a whole dataset of problems, all the histogram bars will look the same for different sample path indices. So having 5 separate paths doesn't seem to add anything – why not just collapse them all into a single graph with three bars (accuracy, precision, recall)?

Author response:
In Figure 6a, the five paths graphed results of five instances of the same experiment, originally intended to demonstrate the robustness of the performance. The dataset is MATH. However, we acknowledge that the graph may be confusing to readers. We have simplified the figure to show only the average of the five instances in the revised Figure 6a, enhancing its clarity. We also changed the caption to "(a) The precision, recall, and accuracy on the MATH dataset. (b) The Acc on MATH in response to the number of sampled reasoning paths when the weight is set to different values" for clarity.

Reviewer comment (Evaluation):
Consider weakening the statement "Each line in Fig. 5 has an obvious trend of going upwards, proving that the increase of Code Usage Frequency induces a general improvement in accuracy": *Proves* is a strong word there, when it's just a nice correlation (not causal – there are many ways of including meaningless code in an output that would not induce an improvement in accuracy). Just state it as a correlation and that's good.

Author response:
**A4**: Thanks for your reminder. We have modified the sentence to "Each line in Fig. 5 has an obvious upward trend, suggesting a possible positive correlation between Code Usage Frequency and accuracy".