Paper: DIVERSITY OF THOUGHT IMPROVES REASONING ABILITIES OF LARGE LANGUAGE MODELS (ICLR-2024)

Reviewer comment (Novelty):
The novelty of the work is very limited. There exists work in which diverse prompts are used to generate diverse reasoning paths: [This paper ](https://arxiv.org/pdf/2206.02336.pdf) should be cited due to its similarity.

Author response:
We thank the reviewer for sharing this paper, but would like to strongly stress that the methodology of this paper is very different from that of ours. In the paper by Li et al., the reasoning problem’s prompt is formed by providing multiple, yet different examples depending on the test sample (refer to figure 10 in their paper), and using a step-wise verifier to correct errors made. **This requires them to have a base of examples from which they can sample few-shot examples for each prompt, a critical limitation as noted in Appendix B.1 of their paper. We do not.**

The “diversity” in their prompts are reasoning paths based on self-consistency for examples from the base of examples (i.e., a hold out set); these reasoning paths are not truly diverse as they all use the same approach (as highlighted in response to reviewer PAGk) and only serve as “more examples” for the model or at best as linguistic diversity triggers (but not reasoning diversity). In contrast, our work (i) involves truly diverse (refer to response to reviewer PAGk) and “related” approaches to solve the reasoning problem (refer to Figure 2 in our work), whilst (ii) limiting the number of examples in the prompt (we use 5, Li et al. need close to 20) and (iii) do not have a step-wise verifier (which further increases the cost).

Reviewer comment (Evaluation):
It is not completely clear why DIV-SE works well on Blockworlds and AQUA-RAT, but less so on GSM8K and Commonsense QA.

Author response:
We would like to stress that the minor improvements are in scenarios where the models are already very performant in the tasks; regardless - the performance using our method is consistently high (with an average improvement of 8 percentage points) and often state-of-the-art (as in the Blocksworld case, with a 29.69 percentage point improvement). All this, with a fraction of the cost of contemporary approaches. We request the reviewer to look at the table below (in the summary link) to learn more about performance gains, and Figure 1 + 3 in the paper to learn more about the cost vs. performance trade-offs. 

The reasons for little improvements for some models and datasets is because of task saturation (particularly in the case of GSM8K and GPT-4). This can be for a variety of reasons, such as these tasks not needing substantial reasoning capabilities as the models become bigger, or dataset contamination. However, for tasks such as Blocksworld, reasoning is more important and this is where our method shines compared to prior approaches.

Summary of the results can be found here:: https://ibb.co/68PF2Dr

Reviewer comment (Experiments):
IDIV is likely to have significant context length constraints.

Author response:
The prompts proposed in the work by Li et al. are significantly longer (with an average of 6050 tokens per prompt, based on their training data provided in https://github.com/microsoft/CodeT/tree/main/DIVERSE) compared to IDIV-SE (which has an average of 523 tokens per prompt) for the GSM8K task – almost a 12x gap. Additionally, the context length of models is increasing (as shown in most recent model releases), and we believe this should not be an issue. Additionally, we have taken measures to mitigate this potential issue. Specifically, we limit our prompts to a maximum of 3-shots, as opposed to (a) 5-7 shots typically used in standard few shot prompts (Wang et al., 2023), or (b) 20 shots as used by the work of Li et al. (https://arxiv.org/pdf/2206.02336.pdf). Please note the inference cost reported in Figure 1 and 3 accounts for both input and output token costs.

Reviewer comment (Experiments):
It’s not clear whether adding a persona increases the diversity of output. I suspect the persona does not add much to the method. This requires an ablation.

Author response:
As part of our preliminary analysis, we observed that the persona was very important in generating enhanced performance. The work of Salewski et al. that we cite highlights the importance of personas in in-context learning situations.

Reviewer comment (Presentation):
The labels on each of the Figures could be clearer. The color codes are missing.

Author response:
Colours in Figures 1 and 3 are used to distinguish between settings (few-shot-CoT and zero-shot-CoT) and methods (our proposed techniques, DIV-SE/IDIV-SE, and the self-consistency method). We will clarify this better in the draft.
In Figure 1, blue and orange represent the few-shot-CoT settings for DIV-SE/IDIV-SE and the self-consistency method, respectively. Similarly, purple and green are used to represent the zero-shot-CoT settings for DIV-SE/IDIV-SE and the self-consistency method, respectively.

In Figure 3, we use blue to show the few-shot-CoT results of our proposed technique, while orange is used for the few-shot-CoT settings of the self-consistency method. Purple is used to represent the zero-shot-CoT results of our proposed technique for the Blocksworld 3 task.

Reviewer comment (Novelty):
While the proposed method by the authors is clear, it appears the novelty of this work is severely limited (see above).

Author response:
Thank you very much for your feedback. We apologize for not being able to respond to your concerns satisfactorily. 

Our edited paper (which is uploaded) hopefully adjusts the claims in a satisfactory manner. We're curious as to why the reviewer is apprehensive when:

1. Our work is the first to highlight the importance of thought diversity (different from prompt diversity as in Li et al.) in solving reasoning problems. Prior reported works aim to batching multiple queries together (unrelated to reasoning), or utilizing more demonstrations with per-step verification (at substantially higher query cost).
2. The performance numbers we report are often state-of-the-art (SOTA). This is particularly true in cases where datasets were released with explicit claims of being challenging for LLMs.
3. The cost for achieving these SOTA numbers is often much lesser than previously published works. This may result in rapid practical adoption.
4. Our evaluation is comparable to those done in previous prompting papers, which were published. We consistently outperform prior work.
5. We've clarified the difference against papers the reviewers have shared, and will gladly incorporate them into discussion of prior work.

Reviewer comment (Evaluation):
IDIV seems to have a significant context length constraint, and the experiments do not show very convincing improvement across the board.

Author response:
The prompts proposed in the work by Li et al. are significantly longer (with an average of 6050 tokens per prompt, based on their training data provided in https://github.com/microsoft/CodeT/tree/main/DIVERSE) compared to IDIV-SE (which has an average of 523 tokens per prompt) for the GSM8K task – almost a 12x gap. Additionally, the context length of models is increasing (as shown in most recent model releases), and we believe this should not be an issue. Additionally, we have taken measures to mitigate this potential issue. Specifically, we limit our prompts to a maximum of 3-shots, as opposed to (a) 5-7 shots typically used in standard few shot prompts (Wang et al., 2023), or (b) 20 shots as used by the work of Li et al. (https://arxiv.org/pdf/2206.02336.pdf). Please note the inference cost reported in Figure 1 and 3 accounts for both input and output token costs.

Reviewer comment (Experiments):
The method requires a held out set where different personas and approaches can be tested to pick useful ones. This makes the proposed DIV-SE method somewhat limited in practical applications. CoT prompting on the other hand provide universally applicable prompts, and self-consistency provides a generally applicable framework without needing a held-out set to fine-tune on.

Author response:
We agree with the observation made by the reviewer that in practice, our proposal requires a hold out set for evaluating the choice of approaches and personas. However, this set turns out to be very small, often with 100 samples (though we believe fewer, e.g., 50 might also suffice). Prompt engineering is typically a one-time effort, and the resulting prompt could potentially serve thousands or even millions of user queries

In addition, all prompt engineering practices in general (and not only our proposed approach) **do** require a hold out set, but the main issue with most practices in the status quo is that they require a significant amount of trial and error (https://docs.anthropic.com/claude/docs/optimizing-your-prompt).

Additionally, the approaches discovered by our DIVERSE-PROMPTING strategy are general and re-usable across reasoning problems of the same domain, adding to the practicality of our approach. Thus, the cost of evaluating the approaches on a hold out set is further amortised. As an example, for the GSM8K task, some of the LLM-generated approaches include: using visualisations, working backwards, using direct calculation, and method of elimination; these can be reused for other arithmetic reasoning problems.

Reviewer comment (Evaluation):
Results seem mixed: Blocksworld clearly benefits from the diversity in prompting. It is clear on this benchmark that with IDIV-SE more can be achieved in less cost. But on other benchmarks this is not necessarily true e.g. GSM8K (Fig 3 & Fig 4).

Author response:
We would like to stress that the minor improvements are in scenarios where the models are already very performant in the tasks; regardless - the performance using our method is consistently high (with an average improvement of 8 percentage points) and often state-of-the-art (as in the Blocksworld case, with a 29.69 percentage point improvement). All this, with a fraction of the cost of contemporary approaches. We request the reviewer to look at the table (in the summary link below) to learn more about performance gains, and Figure 1 + 3 in the paper to learn more about the cost vs. performance trade-offs. 

The reasons for little improvements for some models and datasets is because of task saturation (particularly in the case of GSM8K and GPT-4). This can be for a variety of reasons, such as these tasks not needing substantial reasoning capabilities as the models become bigger, or dataset contamination. However, for tasks such as Blocksworld, reasoning is more important and this is where our method shines compared to prior approaches.

Summary of the results can be found here: https://ibb.co/68PF2Dr

Reviewer comment (Evaluation):
No qualitaitve/quantitative assessment of the synthetically generated demonstrations (using LLMs). For augmented demonstrations - how is the quality of this dataset ensured? I suspect some human annotation to improve the quality of these demonstrations can improve eventual model performance.

Author response:
In generating the demonstrations for a specific approach, we do not solely rely on the LLM to create the demonstrations from scratch. Instead, we utilize the same chain-of-thought demonstrations from Wei et al. (2022) as a base, and instruct GPT-4 to perform a “style transfer” of the solution using the specific approach. 

To ensure the quality of these demonstrations, we conducted a manual evaluation as well. Given that the task primarily involves a style transfer of the demonstrations, we found no errors in the style transferred demonstrations i.e., correctness was preserved.

We also agree with the reviewer that more curation of the “augmented examples” can boost performance. However, we would like to point out that despite the lack of such curation, our approaches demonstrate SOTA performance on tasks that require complex reasoning (e.g., Blocksworld).

Reviewer comment (Experiments):
Limited evaluation: Some commonly used reasoning benchmarks are not present, making it hard to compare with prior work like Wang et al's Self Consistency baseline. Arithmetic reasoning: AddSub MultiArith ASDiv SVAMP. Commonsense and symbolic reasoning:  CSQA StrategyQA ARC-e ARC-c Letter Coinflip

Author response:
Our strategy in selecting the benchmarks was twofold. Firstly, we aimed to choose a diverse set of tasks that required complex reasoning. To this end, we selected tasks involving arithmetic reasoning, algebraic reasoning, commonsense, and planning. Secondly, within each reasoning task, we sought to choose the most challenging benchmark. As shown in Wang et al., 2023, Table 2: AQuA, GSM8K, and CSQA are among the most challenging benchmarks in their respective categories. Similarly, as demonstrated in Valmeekam et al., 2023, the planning benchmark is one of the most challenging for LLMs to solve.

Reviewer comment (Experiments):
Fig 4 & 5 - For a fair comparison with DIVSE-1 to 5, shouldn't Self Consistency baseline also be given multiple reasoning paths instead of just report SC-1?

Author response:
The purpose of Figures 4 and 5 is not to compare the performance of these methods directly, but rather to illustrate how the performance of our proposed techniques varies with the ensemble size. These figures provide a comprehensive comparison, showing the performance of the self-consistency baseline and our proposed techniques under the same conditions. Direct comparison between the self-consistency baseline and our proposed techniques is shown in Figures 1 and 3. Here, self-consistency is given as many as 40 reasoning paths (in AQuA), and we note that our approach provides more utility at a lower cost.

Reviewer comment (Experiments):
The study could also be extended to other models with smaller sizes e.g. Llama 2 7B variants.

Author response:
When evaluating our approach on smaller models (e.g. LLAMA-2 7B), especially if the model is not instruction fine-tuned, its ability to stick to the specification and instructions provided (in generating responses) is very poor. Parsing the results would require significant manual annotation. Additionally, techniques such as chain-of-thought prompt and those that promote multi-step reasoning are most beneficial at models of larger size (https://arxiv.org/pdf/2206.07682.pdf), which is why we also study larger models.

Reviewer comment (Experiments):
What does it mean to have a zero shot setting of DIV-SE? Isn't it always atleast one shot as shown in Fig 2? I couldn't find details of how this is done in zero-shot setting. Is the prompt appended with list of approaches but no examples?

Author response:
In the zero-shot setting of DIV-SE, we indeed provide no examples. Instead, we only append the name of the approach to the prompt. For instance, the prompt might be appended with “Approach: Using algebra”. Figure 2 illustrates the few-shot setting. In all our experiments, we consistently use the same few-shot examples as those used in the study by Wang at el., 2023.

Reviewer comment (Experiments):
What is the quality of these augmentations? Are they really diverse? Are each of them valid? If yes, are they different from other prompts?

Author response:
We have manually evaluated all augmented demonstrations to ensure their quality and found no errors. Regarding diversity, as illustrated in Figure 2, the methodology used in the prompts is indeed diverse. 

When self-consistency is run (for the prompt in figure 2), the responses are below:

Run 1: The candle burns for 5:00 PM - 1:00 PM = 4 hours. Since it melts by 2 centimeters every hour, it will be 4 * 2 = <<4*2=8>>8 centimeters shorter after burning from 1:00 PM to 5:00 PM.

Run 2: The candle burns for 5:00 PM - 1:00 PM = 4 hours. It melts by 2 centimeters every hour, so after 4 hours, it will be 4 * 2 = 8 centimeters shorter.

Run 3: The candle burns for 5:00 PM - 1:00 PM = 4 hours. Since it melts by 2 centimeters every hour, the candle will be 4 * 2 = 8 centimeters shorter.

This clearly highlights the lack of diversity in the approaches.
Can the reviewer suggest a method that we can utilize to quantitatively measure diversity?

Reviewer comment (Experiments):
Sec 2.2 . In practice, for all (persona, approach) combinations, we evaluate the prompt formed using the composition on a held-out set and choose those with the highest performance. This seems like a limitation - for open ended reasoning tasks that could appear in conversational applications, how can this be done outside the benchmarks studied?

Author response:
We agree with the observation made by the reviewer that in practice, our proposal requires a hold out set for evaluating the choice of approaches and personas. However, this set turns out to be very small, often with 100 samples (though we believe fewer, e.g., 50 might also suffice). Prompt engineering is typically a one-time effort, and the resulting prompt could potentially serve thousands or even millions of user queries

In addition, all prompt engineering practices in general (and not only our proposed approach) **do** require a hold out set, but the main issue with most practices in the status quo is that they require a significant amount of trial and error (https://docs.anthropic.com/claude/docs/optimizing-your-prompt).

Additionally, the approaches discovered by our DIVERSE-PROMPTING strategy are general and re-usable across reasoning problems of the same domain, adding to the practicality of our approach. Thus, the cost of evaluating the approaches on a hold out set is further amortised. As an example, for the GSM8K task, some of the LLM-generated approaches include: using visualisations, working backwards, using direct calculation, and method of elimination; these can be reused for other arithmetic reasoning problems.

Reviewer comment (Presentation):
Fig 6 & 8 are referenced in the text but missing in the paper? Did you mean Fig 2?

Author response:
Figure 6 and 8 are presented in the supplementary material. We will modify the text accordingly.

Reviewer comment (Novelty):
Can you explain how different this approach is from self-consistency: I'm not sure if the approach can be described as one bringing diversity in prompting. The diversity is being attempted in the set of approaches taken by the LLM to solve a problem. This is still very similar in spirit to self-consistency - where instead of naively sampling n times, the model is prompted to generate different approaches. The approaches themselves are model generated.

Author response:
The key difference between the two approaches is the nature of the solution paths. Using terminology from our work, the self consistency paper always uses the “same approach” but relies on breaking the problem into smaller subproblems within the approach. However, our work proposes using “different approaches” altogether to solve the problem. This is further highlighted by the fact that the augmentations used in our work are diverse. Additionally, our approach can easily incorporate self consistency within each of the approaches we propose, but the inverse is not true. Most importantly, for a fixed inference budget, our proposed technique outperforms the self consistency method. This indicates that introducing diversity at the conceptual level through various approaches and personas results in more diverse solutions than simply sampling multiple reasoning paths at decoding. **In many ways, diversity-of-thought increases the opportunities for the model to be accurate by purposefully guiding it to use approaches that are relevant to the problem and different from each other.**