Paper: Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Models (ICLR-2024)

Reviewer comment (Novelty):
Although this article proposes several practical strategies, its reasoning framework remains inherently reliant on the Tree-of-thoughts model, thereby limiting its novelty. BoT's structure is restricted to binary trees. Expanding to more complex graph structures will further improve reasoning but is not explored.

Author response:
First, the goal of BoT is not to extend Tree-of-Thoughts reasoning. Nor does it rely on 'Tree of Thoughts' or binary trees. Rather, our contribution lies in a novel way of prompting LLMs by enlisting iteratively accumulated trial-and-error analysis, i.e., 'experiences', into the prompting for LLM in order for it to find the right solution to complex mathematical problems. Prior to this work, CoT or ToT [1] still heavily relied on manually designed examples as demonstrations or human priors as steps for reasoning. The fundamental insight of our paper is that without introducing human annotations or examples, a simple prompt can be refined over iterations by adding 'experiences', which contain the error analysis obtained from previous reasoning steps, to enhance thought generation until a final answer to the complex mathematical problem is attained. Using binary trees is just a vehicle for reasoning step (thought) exploration, and certainly, other CoT schemes can be employed instead to generate intermediate reasoning steps. Therefore, in the paper, our emphasis is on collecting an ensemble of trial-and-error reasoning experiences to be used in promoting LLMs, just like a human with a general math background, when approaching a tough math problem, still needs to tentatively tries to give some answers but learn from trial-and-errors to avoid the pitfalls until the right path to solving the problem is found. By doing so, we open up a new research direction to not extend the prompt with human priors like existing CoT work does but instead focus on how to generate effective error analysis to be fed back into LLMs so that it can find the right path to solve challenging mathematical problems. 

The base thought structure of BoT is not limited to ToT, as we have emphasized in the revised related work and Section C of the Appendix. We choose ToT due to the availability of high-quality source code and its suitableness as a tool to explore the space of reasoning steps in mathematical problems. Although Graph of Thoughts (GoT) [2] can certainly be another natural choice for thought structure, we did not employ it partly because the source code of GoT became publicly available only recently. Another thing is that the GoT paper didn't report its performance on mathematical problems, while the ToT paper did report its performance on Game of 24. However, ToT itself can hardly be extended to other maths problems because of the hardness of coming up with human priors in prompts. In fact, in BoT, we do not recommend complicating the reasoning step generation through GoT or other more advanced thought structures, because we have shown through experiments that for the purpose of thought exploration, ToT, and binary trees in particular, is sufficient. The reason is that if shallow binary trees are used as the base structures, in each iteration, we can create an ensemble of simple and heterogeneous trees for thought structures to well explore the reasoning space.


[1]. Yao, Shunyu, et al., Tree of Thoughts: Deliberate Problem Solving with Large Language Models, Arxiv 2023.

[2]. Besta, Maciej, et al., Graph of thoughts: Solving elaborate problems with large language models, Arxiv 2023.

Reviewer comment (Evaluation):
For analysis, the prompts used to seed BoT could introduce biases and variances. More evaluations on OOD data would be useful to assess the robustness and generalizability of the improvements.

Author response:
BoT is an automated prompting framework, which, starting from a simple prompt (which may be invalid) with no manually-designed demonstrations, iteratively accumulates a history of trial-and-error reasoning experiences toward the right path to problem solving. That is, even if the prompts used to seed BoT could introduce biases or are simply wrong, the thought exploration (via ensembles of binary trees) and error analysis report generated by BoT will be iteratively fed back to prompt the LLM, such that collectively when trial-and-error history is used in prompting the LLM again, the LLM already learns to avoid those pitfalls (poor initial reasoning steps). Therefore, BoT is naturally robust to variation in the initial seed prompt, which does not matter to BoT. In contrast, CoT and ToT traditionally may be affected by human examples and demonstrations or human-suggested thought structures.   

For this reason, without assuming any prior knowledge of the prompt (without assuming any prior derived from any dataset), BoT is directly evaluated on the test set of each dataset. Therefore, there is no OOD issue because there was no training data; all data are assumed unseen and are test data to directly test the generalizability of BoT. OOD evaluation is also less of a concern in existing literature, e.g., ToT, PHP, Complex-CoT.

Reviewer comment (Experiments):
The evaluations are mainly limited to mathematical reasoning. For generality, testing BoT's performance on other domains like commonsense reasoning or symbolic reasoning is needed.

Author response:
We have explicitly mentioned that BoT is meant to solve complex mathematical problems with LLMs starting from the abstract. LLMs are generally believed to be able to chat and perform natural language tasks such as summarization. The specific research question asked by this work is--can LLMs solve complex mathematical problems especially without human annotations or demonstrations? This question is very challenging. 

To review the experimental efforts, the experiments in our manuscript cover 4 commonly used benchmark math datasets and also introduce a more challenging task known as the 'Game of 24,' which even the latest GPT-4 struggles to solve. In the results on MATH shown in Section F of the Appendix, BoT achieves state-of-the-art performance on all categories of the test set.

All these experiments help to verify the capability of BoT to solve complex mathematical problems, which is challenging to achieve, while using BoT toward other reasoning tasks, such as symbolic reasoning, is an interesting direction for future investigation.

Reviewer comment (Experiments):
In the 'Competitors' paragraph, authors mentioned incorporating CoT-SC and Complex CoT as baselines, yet CoT-SC is not shown in the 'mathematical reasoning' part. I hold the view that comparing the proposed method with prevailing baselines like SC(5) or SC(10) will offer a more direct reflection of BoT's efficacy. If it can outperform Complexity-based SC with fewer resources, it would make the work more solid.

Author response:
Because self-consistency (SC) is very resource-consuming as it generally requires sampling a large number of reasoning chains from the LLM, we only adopted CoT-SC as a baseline in the 'Game of 24'. On this task, CoT-SC (k=100), although using the majority output from the $k=100$ best samples, only achieves $9\%$ solving rate. In contrast, BoT achieves $83.7\%$ solving rate, which is a much higher solving rate, without even using SC. This significant gap shows that using SC is only marginally helpful for Game of 24, and the proposed BoT method without relying on SC-based sampling is sufficient to achieve state-of-the-art performance. Therefore, we did not include SC-related mechanisms in the 'mathematical reasoning' part due to its limited improvement in BoT and its huge resource consumption. Besides, existing results in Tables 1, 2, and Section F of the Appendix support our claim that BoT, an automated prompting framework without relying on human annotations, is able to outperform existing state-of-the-art methods, especially PHP+Complex-CoT, by a substantial margin in all problems. While BoT is already significantly better than the current best methods, performing resource-consuming SC-related experiments does not produce further evidence of the excellence of BoT, especially considering CoT-SC (k=100) lags behind BoT substantially in the 'Game of 24'. In fact, our proposed BoT methods offer different insights than SC. We show that there is a way to generate and select thoughts (as intermediate reasoning steps) and feed the analysis on these thoughts done by the LLM back into LLM prompting in order to find the right path to problem solving. 

In the 'mathematical reasoning' task, no method, including Complex-CoT and PHP Complex-CoT, utilizes the SC ensemble mechanism. The prompt of these methods simply includes the reasoning examples derived from the Complex CoT. Especially, PHP utilizes the greedy decoding (i.e., temperature = 0). We have provided more details in Section F of the appendix. In the 'Game of 24' task, CoT-SC (k=100) takes the majority of output from $100$ reasoning chains.

Reviewer comment (Experiments):
The study conducts experiments based on GPT-4, which can lead to substantially high experimentation costs. Have the authors considered or utilized more cost-effective options like GPT-3.5-turbo? I'm aware of the recent variability in performance of this model. However, if there are experimental results showing that GPT-3.5 combined with BoT can outperform GPT-4 with CoT/CoT-SC, it would render the study's findings more convincing.

Author response:
Following the reviewer's suggestions, in Section F of the appendix, we have added more experiments on the MATH dataset during the rebuttal period. The Table below presents the solving rate ($\%$) in all samples (Overall) and the category Precalculus for different model-method combinations. We refer the reviewer to Section F of the appendix for more details. 

| Methods             | Overall | Precalculus |
|---------------------|---------|-------------|
| GPT3.5 ComplexCoT   | 34.1    | 14.5        |
| GPT3.5 BoT          | 40.61   | 15.5        |
| GPT4 ComplexCoT     | 50.3    | 26.7        |
| GPT4 PHP+ComplexCoT | 53.9    | 29.8        |
| GPT3.5 BoT (GPT4)   | 55.8    | 27.9        |

"if there are experimental results showing that GPT-3.5 combined with BoT can outperform GPT-4 with CoT/CoT-SC, it would render the study's findings more convincing."
This is simply not possible and is not a fair comparison. On the MATH dataset, the solving rate of GPT-3.5-Turbo + BoT (GPT3.5 BoT) turns out to be not as good as GPT4 + ComplexCoT. This is not a fair comparison anyway, just because GPT4 is way more powerful than GPT3.5, which is evidenced by the fair comparison between GPT3.5 + ComplexCoT and GPT4 + ComplexCoT. This result is offered here just out of curiosity reasons. However, GPT3.5 + BoT is much better than GPT3.5 + ComplexCoT, showing BoT is a better method than ComplexCoT as a fair comparison.

However, what is interesting is that if GPT4 is used to generate experience while GPT3.5 is used for thought generation, referred to as GPT3.5 BoT (GPT4), the solving rate for BoT increases to $55.8\%$, which is not only $5.5\%$ higher than GPT4 + ComplexCoT but also outperforms the current state-of-the-art GPT4 + PHP+ComplexCoT by $1.9\%$. This is an important finding as it shows that the performance of BoT actually depends more on the quality of experiences, and even using the inferior GPT3.5 for thought generation, BoT is a powerful method and can still beat PHP+ComplexCoT using GPT4.

Thus, a viable resource-efficient way could be to use resource-friendly LLM, such as GPT-3.5-Turbo, for thought generation, while only using the more powerful GPT4 for error analysis to generate experiences as guidance. Such a combination can achieve a better tradeoff between performance and resource consumption, according to the Table.

Overall, we are committed to addressing reviewer questions and concerns and improving the quality of this research. It is much appreciated if the reviewer could reassess the quality of this work based on the value of the research question it is addressing, its novelty and the outstanding performance in comparison to SOTA methods in the literature.

Reviewer comment (Presentation):
The manuscript needs polishing in their figures' presentation, e.g., the authors need to give more detailed examples in Fig1.

Author response:
Thanks for your comments and we reply to them below.

__Response to W1&Q1__: 

We have revised and polished all Figures, especially Figure 1, in the submission. To provide further clarity, additional details about each component of BoT, as illustrated in Figure 1 and Figure 2, have been included in sections A, C, and D of the appendix. In particular, we have supplemented the appendix with additional experimental results, including the experience in Table 7 and Table 8, generated by BoT. Furthermore, we have enhanced the organization of the source code within the _examples/BoostingOfThought/_ directory to provide a more comprehensive overview of the BoT implementation. For instance, within the code file _BoT\_commenter.py_, one can find the process for generating experience using LLMs, while _BoT\_reasoner.py_ details how this experience is structured and incorporated into the prompt for subsequent iterations of reasoning.

Specifically, both the 'error input' and 'experience' are included in the prompt, but the 'error input' is embraced as the sub-block of 'experience'. In fact, the experience comprises input reasoning steps, the conclusion, error analysis, advice, and the confidence score, all of which are generated automatically by LLMs. As each iteration of BoT produces one such experience for the reasoning chain, LLMs can gradually generate the right answer due to the experience accumulation in the prompt over iterations. Tables 7 and 8 of the appendix provide a direct example of how experience is organized in the prompt. 

The confusion arising from Figure 1 primarily stems from our intention to first emphasize the significance and effectiveness of an ensemble of trial-and-error reasoning experiences within the context of the proposed automated prompting framework BoT. To mitigate the confusion, we have incorporated additional explanations into Figure 1.

Reviewer comment (Reproducibility):
In the prompt, it is unclear whether the “error input” is included, or only the “experience” is included. From figure 1, only an “error report” like “step 1 is not closer to 24” is shown, not the “error input” such as what is step 1, 2, 3. It is unclear how the LLM knows what step 1 means, and how it can learn from the error if it does not know the specific input.

Author response:
Thanks for your comments and we reply to them below.

__Response to W1&Q1__: 

We have revised and polished all Figures, especially Figure 1, in the submission. To provide further clarity, additional details about each component of BoT, as illustrated in Figure 1 and Figure 2, have been included in sections A, C, and D of the appendix. In particular, we have supplemented the appendix with additional experimental results, including the experience in Table 7 and Table 8, generated by BoT. Furthermore, we have enhanced the organization of the source code within the _examples/BoostingOfThought/_ directory to provide a more comprehensive overview of the BoT implementation. For instance, within the code file _BoT\_commenter.py_, one can find the process for generating experience using LLMs, while _BoT\_reasoner.py_ details how this experience is structured and incorporated into the prompt for subsequent iterations of reasoning.

Specifically, both the 'error input' and 'experience' are included in the prompt, but the 'error input' is embraced as the sub-block of 'experience'. In fact, the experience comprises input reasoning steps, the conclusion, error analysis, advice, and the confidence score, all of which are generated automatically by LLMs. As each iteration of BoT produces one such experience for the reasoning chain, LLMs can gradually generate the right answer due to the experience accumulation in the prompt over iterations. Tables 7 and 8 of the appendix provide a direct example of how experience is organized in the prompt. 

The confusion arising from Figure 1 primarily stems from our intention to first emphasize the significance and effectiveness of an ensemble of trial-and-error reasoning experiences within the context of the proposed automated prompting framework BoT. To mitigate the confusion, we have incorporated additional explanations into Figure 1.

Reviewer comment (Novelty):
Could the authors consider the entire (input, error analysis) as an In-context Learning example? In that case, the entire method would be similar to CoT, meaning that one could manually construct an exemplar consisting of an (input, error analysis) pair, then use the CoT idea to follow the strategy to generate analysis and think about the correct answer.

Author response:
__Response to Q2__:

Your further suggestion to include manually constructed (input, error analysis) as an in-context learning example in the prompt, similar to the CoT approach, is highly insightful. We also contemplated this intriguing idea but had to abandon it due to three key concerns. First, relying on human priors to manually design error analysis and advice is time-consuming, inefficient, and hard to generalize well to different tasks. Second, BoT, an automated prompting framework with experience generated by LLMs, is able to achieve a competitive or even top problem-solving rate on multiple mathematical problems, as shown by experimental results. Third, embracing human knowledge in the prompt may have limited applications in real-world usage due to possible bias, misleading, and security issues brought by others.

Reviewer comment (Novelty):
I agree that prompt engineering is crucial for LLM applications. However, it's worth noting that prompt engineering is often model-dependent, and the techniques may evolve as LLM capabilities improve. This may not offer long-term guidance for research unless it uncovers fundamental insights. This distinction is critical in differentiating academic research from practical production. Therefore, while the paper does offer valuable techniques for prompting the model and achieving good results on evaluation sets, it lacks in-depth discussion of the underlying reasons. This makes the paper better suited for application-oriented conferences rather than ICLR.

Author response:
First, we cannot agree with the reviewer that this work does not offer long-term guidance for research and does not uncover fundamental insights. We do have offered fundamental insights for reasoning with LLMs. Traditionally, LLMs cannot solve complex mathematical problems sufficiently well. We show that there is a way to generate and select thoughts, and leverage the error analysis generated by LLM itself for these thoughts and feeding such error report back to prompt the LLM, we can progressively and accumulatively derive the right prompt required for LLM to generate the solution to complex mathematical problems. It shows the latent capability and potential of LLM can be unleashed through proper triggering from the error analysis and advice on the generated and selected reasoning steps. This process is fundamentally similar to humans who possess basic mathematical skills and are tasked to solve a math question. For solving a particular question, one sometimes must learn from not only demonstrations but also prior trials and errors to progressively discover the reasoning path. Intuitively speaking, Chain-of-Thoughts (CoT) [1] shows that demonstration via examples is conducive to problem solving. However, we point out that demonstration is not enough and also hard to obtain anyway. One must also practice and encounter errors and absorb such trial-and-errors into his/her knowledge base to solve the problem. So does an LLM. Thus, BoT offers fundamental insights and guidance on how to enable LLMs to generate effective reasoning steps for complex problem solving, essentially through retrospective error analysis, without requiring human demonstrations.

With these insights, the research of prompt engineering on inducing the reasoning capability of LLMs can focus on how to generate a compiled list of trial-and-error history (error analysis report from solving the problem) instead of introducing more human priors (examples) to the prompt like existing CoT, ToT methods would do, which we believe is both insufficient and hard to acquire. This, in turn, makes BoT an automated prompting framework because by iteratively collecting effective error analysis in the prompt without human annotations, LLMs can be guided to produce a correct reasoning chain toward problem solving. 

Second, our insight is NOT model dependent---the same method can be successfully applied to several LLMs, such as gpt-3.5-turbo, gpt-4, and Llama2, according to our experiments. In particular, we utilize the gpt-3.5-turbo model to generate examples shown in the Appendix. The results still support our insights and conclusion. In Section B of the Appendix, we present more discussions on the fundamental insights of BoT. Furthermore, the base model in BoT's boosting mechanism does not have to be specific tree thought structures, such as ToT [2]. We chose ToT here for its effectiveness and simplicity. Therefore, the fundamental concept underpinning the proposed BoT's outstanding performance with an automated prompting framework lies in recognizing the fundamental cycle of thought generation, selection and error analysis to be fed back into LLM prompting as advice and guidance, which unlocks LLM's ability to solve complex problems. We propose BoT as a general trial-and-error prompting framework to enhance the reasoning capabilities of LLMs (which proves to be effective on multiple LLMs in general), rather than a model-dependent engineering trick that will fade away as models evolve.  

Third, we are certainly not targeting applications or practical production related to LLMs. While most real-world LLM applications are regarding chatting with humans (i.e., ChatGPT), it is an emerging research question arising in the NeurIPS/ICLR/ICML/AAAI community to ask whether LLMs can solve complex mathematical questions (which is certainly not applications today, but simply a research question asked by the research community out of pure intellectual interests). For example, Chain of Thoughts (CoT) arises in NeurIPS 2022. To answer this fundamental question, we show the possibility of prompting LLM in a different way that no other work on Chain of Thoughts has tried out before, i.e., through a log of trial-and-error analysis, and show that this style of prompting (instead of just demonstrating human priors to LLMs) is key to unlocking LLMs' capabilities to solve complex mathematical problems. The underlying reason is that simply demonstrating to LLMs the steps to solve a math problem is not enough. We must also leverage a compiled list of trials and errors obtained from attempting to solve the problem so that the LLM progressively learns to avoid pitfalls, just like humans. Our experiments have substantiated our claim by showing that without human priors, BoT can achieve outstanding performance at solving various mathematical problems via the proposed thought generation, aggregation, and error analysis accumulated to be used for prompting LLMs.

Reviewer comment (Evaluation):
LLMs can be unstable and prone to hallucination, which could result in bad or incorrect feedback when using the Boosting of Thoughts (BoT) iterative prompting framework. Is there analysis on the impact of "bad" LLM feedback? Further, as the iterative produces are automatic, spurious feedback could get amplified over iterations. Some discuss may be necessary.

Author response:
In Section E of the appendix, we have discussed that the spurious feedback may lead the LLMs to generate reasoning steps that are logically incorrect or do not adhere to any of the task rules. Table 7 of the Appendix presents the corresponding results of BoT when the spurious feedback is included as the experience in the prompt. However, we argue that spurious feedback will NOT be amplified over iterations; instead, thanks to the iterative mechanism of BoT, its negative impact on the generated reasoning steps can be mitigated or even entirely rectified in subsequent iterations. Specifically, as the wrong reasoning steps caused by spurious feedback contain obvious mistakes, it is likely that LLMs tend to generate correct error analysis by comparing to the final target at some later steps and provide effective suggestions for revisions. With this new experience included in the prompt, BoT is capable of generating correct thoughts (reasoning steps). This is further mitigated by the proposed thought generation process through an ensemble of binary trees and aggregating them to acquire the better chain of thoughts to be used in error analysis in each iteration.  
As demonstrated by the experience in Table 8, BoT produces detailed error reports and revision suggestions, resulting in a rational thought generation process illustrated in Table 7 of the Appendix. 

The advantage of BoT, which leverages iterations to mitigate the detrimental effects of invalid or erroneous feedback (i.e., the LLMs progressively avoid pitfalls during the BoT process), is evident in Figure 4. Notably, the performance of BoT exhibits consistent enhancement as the number of iterations increases. This implies both the significance of accumulating trial-and-error experiences iteratively and the capacity of subsequent experiences to rectify (avoid) errors in earlier experiences.

Reviewer comment (Experiments):
Details are lacking on key components like aggregation strategies and generating edge weights for trees. More analysis or ablation studies are also helpful.

Author response:
We have added more details and discussions on edge weight computation for trees and the aggregation strategies in Sections C and D of the Appendix. In summary, to calculate the edge weight $V_{i-1, i}$ between two reasoning steps, represented as nodes in the tree and denoted by $z_{i-1}$ and $z_{i}$, LLMs are utilized to evaluate the entire reasoning chain $z_{1, ..., i-1, i}$, which forms a branch of the tree with $z_1$ acting as the root node. The corresponding prompt used by LLMs for this weight computation can be found in Section A of the Appendix and the source code _examples/BoostingOfThought/BoT\_reasoner.py_.

Reviewer comment (Reproducibility):
In Section 3.2, it is not quite clear how to calculate the weights for the weighted binary tree.

Author response:
We have added more details and discussions on edge weight computation for trees and the aggregation strategies in Sections C and D of the Appendix. In summary, to calculate the edge weight $V_{i-1, i}$ between two reasoning steps, represented as nodes in the tree and denoted by $z_{i-1}$ and $z_{i}$, LLMs are utilized to evaluate the entire reasoning chain $z_{1, ..., i-1, i}$, which forms a branch of the tree with $z_1$ acting as the root node. The corresponding prompt used by LLMs for this weight computation can be found in Section A of the Appendix and the source code _examples/BoostingOfThought/BoT\_reasoner.py_.