Paper: Cumulative Reasoning with Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
The main weakness of the paper is that it's unclear where exactly the contribution comes from: the ability of the LLM to propose but then self-correctness? The grounding to First-order-logic (FOL)? Or the wider search among candidates and hypothesis? A thorough ablation study feels very needed.

Author response:
**W1: Clarification of the contribution source in CR**

We conducted an ablation study on FOLIO-wiki to measure the Verifier (which is the self-correctness ability you mentioned) as well as the random choice of premises in the Proposer (affecting the width of search among candidates and hypothesis you mentioned), and the results are shown in the table below:

| Method | CoT | CR | CR(w/o Verifier) | CR(w/o random choice of premises) | CR(w/o Verifier and random choice of premises) |
| --- | --- | --- | --- | --- | --- |
| **Acc.** | 64.61 | 73.03(+8.42) | 64.23(-0.38) | 68.73(+4.12) | 67.23(+2.62) |

It is evident that our Verifier, as well as the random choice of remises in Proposer, are quite effective. At the same time, it is straightforward that if we remove the Verifier but retain the random choice of remises skill in Proposer, the quality of propositions put forward by the Proposer without verification will obviously decrease significantly, which will ultimately have a negative impact. However, when both methods are applied, they enhance the reasoning capability of GPT-3.5-Turbo on FOLIO-wiki in a synergistic way, with an effect greater than the sum of their individual improvements.

What's more, we have the following few points that need to be supplemented：

- In the AutoTNLI experiment, we did not implement the verifier, just concatenate all the generated propositions to form a longer context, then still achieve superior results over Direct, CoT, and CoT-SC, even using relatively small language models such as LLaMA-13B and LLaMA-65B.

- In solving MATH problems without code environment experiments, we still did not implement the verifier LLM to reduce the computation cost, but still achieved SOTA results on this dataset:

> Finally, on the MATH dataset, we establish new state-of-the-art results with 58.0% overall accuracy, surpassing the previous best approach by a margin of 4.2%, and achieving 43% relative improvement on the hardest level 5 problems (22.4% to 32.1%).

- In solving MATH problems with code environment experiments, we also did not implement the verifier LLM to reduce the computation cost (Only one LLM is used, with one consecutive thinking context session), but also achieved SOTA results on this dataset:

> We achieved a 72.2% accuracy on the MATH dataset, significantly outperforming benchmarks like PAL (52%) and ToRA (60.8%). Notably, there was a 66.8% relative improvement over PAL and 12.8% over ToRA on the most challenging level 5 MATH problems, demonstrating the effectiveness of CR in a code environment and further validating the robustness of CR in handling complex tasks.

- In addition, in the successive work DetermLR (https://arxiv.org/pdf/2310.18659.pdf) based on our method CR (their code is mainly based on our released code, with more intricate thinking context management mechanism), the authors did comprehensive experiments on 4 different logical inference dataset, and find that our method and its extension DetermLR,  consistently outperformed the renowned baselines, including CoT, CoT-SC, and ToT, with less computation cost (visited states).

Reviewer comment (Experiments):
It seems that the approach is different in every task: in FOLIO-wiki it relies mostly on FOL deductions, but in the Game of 24 it relies mostly in a wider *search* (like a wide beam search) that allows it to explore more hypotheses than the baseline. Is the proposed approach really reasoning better, or is it just a brute-force exploration? I would like to know which gains come from the actual approach, while all side-improvements like a wider search are equal across all baselines.

Author response:
**W2: CR's adaptation to different tasks**


CR's strength lies in its flexibility to adapt to different tasks by leveraging accumulated historical information and compositional reasoning based on previous results. While the methodology indeed varies with tasks, the core philosophy of CR—cumulative reasoning with validation and reporting—remains consistent. This adaptability is a testament to CR's robustness across varied problem domains.

For example, in the Game of 24, CR's methodology involves a more exploratory approach, which is indeed different from the FOLIO-wiki. However, this does not dilute the reasoning capability of CR; instead, it exemplifies its versatility in addressing diverse problem-solving paradigms.

Reviewer comment (Experiments):
The method, explained in Section 3.1, is a bit tailored to problems that can be described as a series of "predicates" such as the Folio dataset from Han et al. 2022 ("FOLIO: Natural Language Reasoning with First-Order Logic"). I wonder how well it generalizes beyond tasks that are explicitly defined in FOL.

Author response:
**W3: Generalizability of CR beyond FOL-defined tasks**

CR is designed to be a general-purpose reasoning framework. Its application is not limited to tasks explicitly defined in FOL. For instance, in the MATH dataset, which is not based on FOL, CR still demonstrates superior performance. This indicates that CR's methodology, while particularly effective for FOL tasks, is adaptable and efficient for a broad spectrum of reasoning challenges.

Reviewer comment (Reproducibility):
I am missing some actual prompts. It would have been useful to see the actual prompts that were used for each agent, as well as the code.

Author response:
**Q1: Availability of prompts and code**

We have made our code and the prompts used for each agent publicly available at [https://anonymous.4open.science/r/cumulative-reasoning-anonymous-4477](https://anonymous.4open.science/r/cumulative-reasoning-anonymous-4477). This repository includes various few-shot examples for different language models, reflecting their specific roles in the CR framework.
We also present the meta prompt for solving MATH problems, you can just play with it using OpenAI assistant API, we have created a demo using this prompt on chat.openai.com:  https://chat.openai.com/g/g-L3a4ZCIHx-cr-agent-v0-1 .

Reviewer comment (Reproducibility):
How do the prompts actually look like?

Author response:
**Q1: Availability of prompts and code**

We have made our code and the prompts used for each agent publicly available at [https://anonymous.4open.science/r/cumulative-reasoning-anonymous-4477](https://anonymous.4open.science/r/cumulative-reasoning-anonymous-4477). This repository includes various few-shot examples for different language models, reflecting their specific roles in the CR framework.
We also present the meta prompt for solving MATH problems, you can just play with it using OpenAI assistant API, we have created a demo using this prompt on chat.openai.com:  https://chat.openai.com/g/g-L3a4ZCIHx-cr-agent-v0-1 .

Reviewer comment (Experiments):
What exactly is the `n` parameter - the "number of generated intermediate propositions"? Is it like the number of beams in beam search? What happens when `n=1`, does the proposed approach still provide gains over the baselines such as CoT and SC? This is very important to understand, because if the gains are only coming from the wide beam search, maybe the idea can be applied to simpler methods as well.

Author response:
**Q2: Clarification on the parameter 'n'**

In FOLIO tasks, 'n' represents the number of verified generated propositions. When 'n=1' and equipped with verifiers, CR still exhibits gains over baselines such as CoT and SC, indicating that the benefits of CR extend beyond merely a wide beam search approach. It is not the beam search numbers, but the number of accumulative intermediate results, that is, the beam search's results are disjointed, but our thinking context is consecutive, which makes the reasoning process of CR different from CoT-SC and ToT. In AutoTNLI, where we did not implement verifiers, 'n' refers to the number of accumulated generated propositions, and when n grows, the accuracy consistently improves. For more ablations on logical inference datasets, please see Figure 3 (The impact of the number of generated determinate premises) in the paper DetermLR (https://arxiv.org/pdf/2310.18659.pdf), which also shows consistent improvement with more accumulated propositions.

Reviewer comment (Presentation):
Are there examples for predictions in TNLI?

Author response:
**Q3: Examples for predictions in TNLI**

The authors of AutoTNLI have made a brilliant webpage: https://autotnli.github.io/, and according to Table 1 in their paper (https://vgupta123.github.io/docs/autotnli.pdf), they have presented a nice example in Person category, the task is predicting the given hypothesis is True or False (they denoted them as Entail and Contradict).

Reviewer comment (Presentation):
The nice background on Logic in section 2 is clear and nice, but I am not sure how is it needed for reading the paper. Some tasks are based on logic (FOLIO), and in that case, I understand why is it relevant. But when the task is not based on Logic, is the background of Section 2 relevant in any way?

Author response:
**Q4: Relevance of Logic Background in non-FOL-based tasks**


The main theoretical motivation of our method lies in the intuitionistic logic, the philosophy of [mathematical constructivism](https://en.wikipedia.org/wiki/Constructivism_(philosophy_of_mathematics)), and the topos theory, which asserts that the cumulative process of constructing new propositions is the most natural way to perform complex reasoning, especially in the realm of (higher-order) logic and pure mathematics.

The background on logic provided in Section 2 is crucial for tasks based on FOL. For tasks not explicitly grounded in logic, such as the Game of 24 or the MATH dataset, this background serves as a conceptual framework for the reasoning strategies employed by our CR method. It helps contextualize the approach and its potential applications in a broader spectrum of reasoning tasks, including those not strictly defined in FOL, such as the AutoTNLI dataset, which can only be represented in Higher-order Logic, and the MATH dataset, which can only be represented in Categorical Logic and Topos Theory.

Reviewer comment (Presentation):
Most of the tables contain a column for Accuracy *and* another column for Error, where the Error is exactly `100-Accuracy`. This is redundant, and it makes the false impression as if there was an additional metric or twice the number of results.

Author response:
**Additional Comments:**

- **On Redundancy in Tables**: We acknowledge the redundancy of Accuracy and Error columns. In our revised version, we have streamlined these metrics for clarity.

Reviewer comment (Presentation):
The appendix is a bit confusing: some Figures contain the generated solution by both CoT and CR; some datasets have only the input and the ground truth outputs (e.g., Figure 5 and 6 and 7 without any model-predicted solution); while some Figures (e.g. in Appendix C.1) do not contain any caption, so I am not sure I can follow them.

Author response:
**Additional Comments:**

- **Appendix Clarification**: We have revised the appendix to provide clear captions and consistent presentation of model-predicted solutions

Reviewer comment (Novelty):
**Novelty:** The main components of the proposed model (decomposed reasoning, using verifiers, and using a halter model) have been explored in several existing work. The way the components have been combined also resembles some of existing works in the literature. For example, [1] has four components where the Selection and Inference modules can be seen as a variant of the Reporter in this work, the Halter can be seen as a variant of the Reporter, and the Value Function can be seen as a variant of the Verifier (to be clear, I'm not saying the components are identical).

Author response:
**Novelty**

Please see the general response. We believe that compared with the existing methods like Faithful Reasoning, Maieutic Prompting, etc., the main contribution of CR is cumulative reasoning plus asking sub-questions/hints.

Reviewer comment (Evaluation):
**Statistical significance:** Comparing CR and CoT-SC in Table 2, many of the improvements are in the order of 1-2% (with the exception of GPT3.5 results). For prompting techniques, given the high dependence of model performance on the prompts, I'm not sure how statistically significant these improvements are. The small size of the dataset is understandable, but that makes the statistical significance of these reported improvements even more questionable.

Author response:
**Statistical Significance**

Please see the general response. The DetermLR paper has reproduced CR and got consistent outcomes showing that CR is more effective than CoT/ToT with fewer visited states.

Reviewer comment (Experiments):
**Game of 24 results:** I'm concerned about the Game of 24 experimental results. If I understand correctly, your approach for this dataset is almost equivalent to a naive brute forcing over all possible solutions (and I say "almost" because the LLM can of course rule out some of the obviously wrong cases; e.g., if the current state evaluates to 20 and the only remaining number is 4, the LLM probably understands that a summation is needed). It also seems like the model slightly changes here, because now it needs to keep a set of reached states and keep expanding them, as opposed to deriving valid conclusions from the previous premises.

Author response:
**Game of 24 results**

If we do not restrict the number of visited states, our method is indeed a brute force search algorithm for the Game of 24. However, what we did in CR is trying to exploit the preferences of LLM based on the current situation, so that we can get the correct answer by visiting as few states as possible, and avoid unnecessary mistakes. This process is not trivial and differentiates CR with pure brute force algorithm. Indeed, compared with baselines (ToT), CR has higher accuracy and smaller visited states, which is a non-trivial result. Moreover, as you said, our model indeed changes slightly here, where our main modification is setting the current explored state as a "premise", with the rest part unchanged.

Reviewer comment (Evaluation):
What's the average number of model calls needed for CR?

Author response:
**API calls**

See the general response.

Reviewer comment (Reproducibility):
How does the prompt for CR components look like?

Author response:
**CR Prompt**

We have made our code and prompts publicly available at https://anonymous.4open.science/r/cumulative-reasoning-anonymous-4477, which contains all the prompts that we used.

Reviewer comment (Experiments):
Firstly, I find the comparison with the number of samples for CoT somewhat perplexing. While it may appear to be a fair comparison, in reality, CR requires considerably more computational resources (in terms of API calls). For instance, in the case of the Game of 24, the comparison is made with CoT-SC, which employs 16 samples and conducts a majority vote among them. This necessitates 16 API calls to the model. In contrast, CR maintains "n=4," indicating that at each step, the proposer makes four API calls, the verifier makes one call, and the reporter makes another call to determine whether the step constitutes the final answer. Consequently, a total of six calls are made at each step. With a maximum of 50 iterations, the maximum API calls can reach as high as 300. Therefore, for a fair comparison, it is imperative to evaluate CR against "k=300," as CR proves to be considerably more computationally expensive for a relatively marginal performance gain. The paper appears to lack an accuracy versus cost tradeoff graph, which would be instrumental in assessing this aspect. Can the authors please confirm if my understanding is accurate?

Author response:
**Computational Resources**

You are right, in terms of total computational resources, CR is slightly higher than other methods. For example, as stated in General Response, CR with PHP needs 12 API calls, while complex-cot (w/ PHP) uses 7.77 API calls. However, CR explores fewer states compared with CoT/CoT-SC/ToT, which means CR just spends comparatively more time on each state. We'd like to highlight two important aspects in this context:

- **Algorithmic Nature and Resource Utilization**: It is difficult to adjust each algorithm so that all of them use similar number of API calls, because each algorithm inherently varies in its approach to problem-solving. CoT, for instance, employs a straightforward linear thinking process, naturally limiting its time spent per state. On the other hand, ToT involves a breadth-first search, where the extent of exploration is adjustable. CR's slightly higher resource use is a byproduct of its design philosophy, spending more resources on each state.

- **Exploring LLMs' Capabilities**: We are in an era of pushing the boundaries of what LLMs can achieve. Our focus with CR is to test the limits of LLMs in solving complex problems, potentially to a human-equivalent level. This exploration, even with a higher resource footprint, is vital for understanding the full potential of LLMs in tackling challenging tasks.

Reviewer comment (Evaluation):
Furthermore, in this methodology, there is a potential for a substantial amplification of errors, with a significant reliance on the verifier. If a verifier selects one of the reasoning chains proposed by the proposer, all subsequent chains will be built upon that initial choice. It becomes crucial for the verifier to make accurate decisions, particularly early in the decision chain. To gain insights into the potential propagation of errors throughout the chain, it is essential to conduct an error analysis concerning the verifier's accuracy at each step.

Author response:
**Reliance on verifier**

We believe that the success of our method comes from the "cumulative" part, not the verified part. Indeed, in our MATH experiment, CR did not use verifiers, but it still significantly outperformed baseline algorithms. Moreover, in CR, the proposer will not use all the previous steps in each proposal, instead, it will pick a few promising ones. Therefore, the quality of verifier is important, but not crucially important.  This makes the error analysis of verifier's accuracy fairly challenging. For now we can only claim we do not need verifiers in MATH, but we are open to testing this further in other domains.

Reviewer comment (Experiments):
Lastly, it's worth noting that the paper bears a resemblance to numerous prior works, but a comprehensive comparison with them on the same dataset is notably absent. While the authors did compare their approach to PHP and the Tree of Thought method, there are other relevant works to consider. For instance, prior works such as "Faithful Reasoning Using Large Language Models" (Creswell and Shanahan), which follows a similar methodology of select, infer, and halt; "Maieutic Prompting" (Jung et al.), where they generate and prune; "Training Verifiers" (Cobbe et al), which involves generating multiple solutions and ranking them; "LM vs LM" (Cohen et al), in which one large language model can identify consistencies in another, and so on. A comparative analysis on similar datasets could assist the authors in providing more robust justification for their claims.

Author response:
**Compare with other methods**

Thank you for your suggestions on comparing with other methods! In fact, we have considered many of the methods that you mentioned, but most of them are not open sourced (except Maieutic prompting), which makes it difficult to have a fair comparison. We aim to include these baselines in future experiments, subject to resource availability and API access from OpenAI. Moreover, as we mentioned in the General Response, the experimental results from DetermLR paper have supported our method's efficacy.

Reviewer comment (Experiments):
The author should also compare the ToT baseline in the logic reasoning and math datasets since it also involves iterative prompting and is more comparable to the proposed method.

Author response:
**Q1: Comparison with ToT Baseline**

Thank you for highlighting the importance of comparing our Cumulative Reasoning (CR) framework with the ToT baseline. While our initial submission did not include this comparison, recent work that builds on the Cumulative Reasoning (CR) framework, named DetermLR, show that CR outperforms ToT across several datasets, including LogiQA, ProofWriter, and LogicalDeduction, with fewer state visits. These results indirectly validate the efficacy of CR in contexts where ToT is also applicable, offering a broader perspective on CR's capabilities in various reasoning tasks. (Please see the General Response for detailed findings.)

Reviewer comment (Evaluation):
How is the time/compute efficiency of the proposed method compared to the baselines? I understand that the proposed method visited fewer states during inference, but I'm not quite sure if the compute/number of GPT4 prompting for each state visiting is the same as the baselines.

Author response:
**Q2: Time/Compute Efficiency**

Regarding time and compute efficiency, we meticulously controlled the total number of prompts used in CR to ensure it did not exceed a certain multiple of the baselines. Specifically, the total number of invocations in CR is kept within a manageable limit, comparable to other methods. As for the time aspect, it's challenging to provide an exact estimate due to dependencies on the response rate of the OpenAI API, which can vary based on user activity at the time of execution. We observed considerable variance in the server response speeds during our experiments. However, we ensured that efficiency was within reasonable bounds, making CR a viable alternative to existing methods in practical scenarios. We've provided #API calls comparison in the General Response.

Reviewer comment (Reproducibility):
What are the prompts used for implementing the proposer, verifier, and reporter? It is not clear how LLMs achieve those functions.

Author response:
**Q3: Implementation of LLM Roles**

We understand the importance of clarity in how the LLMs are employed for the different roles of proposer, verifier, and reporter in our method. To address this, we have made our code and the prompts publicly available at https://anonymous.4open.science/r/cumulative-reasoning-anonymous-4477. This repository includes detailed examples of the prompts used for each LLM role, providing insight into how these models collectively contribute to the CR framework. We believe this will offer a comprehensive understanding of their functionalities and interactions within our proposed method.