Paper: Reprompting: Automated Chain-of-Thought Prompt Inference Through Gibbs Sampling (ICLR-2024)

Reviewer comment (Theory):
major: the algorithm is designed based on assumptions (1) and (2), which are **not standard** in LLMs.

Author response:
The assumptions in Eq(1)(2) are based on the empirical findings in [1,2] on a wide range of tasks. Specifically, [1] shows that the test performance does not vary much when increasing the number of supervised examples in the prompt from 8 to 32 or more. Chain-of-Thought (CoT) prompting further reduces the number of demonstrations needed -- the test performance remains similar when there is a varying number (but typically >= 3) of few-shot exemplars. These findings can be expressed formerly in mathematical forms as in Eq(1)(2), and have served as the base of LLM evaluation through few-shot prompting in prior work [3,4] . We have updated Section 2.1 to better explain it.

Reviewer comment (Experiments):
major: in experiments, is message $m$ (i.e., *Let’s think step by step. At the end, show your answer bracketed with <answer> and </answer>. Finally generate END at the end of the solution*) applied to all other baselines? If not, an ablation study is required to justify where the improvement comes from, i.e., the message or the proposed reprompting technique.

Author response:
The message “Let’s think step by step…” is applied to all the baselines, except for the few-shot baseline where the demonstrations only include the concatenated question answer pairs without intermediate steps, thus we didn’t include the “step-by-step” instruction in the message.

Reviewer comment (Experiments):
major: experiments are weak: popular benchmarks (e.g., datasets used in Tables 2 and 3 of Self-Consistency) have not been used in the paper.

Author response:
We have added more experiments on 10 additional tasks and popular benchmarks, including 7 tasks from BigBench Hard (BBH), GSM8K (math word problems) and two from Hendrycks’ MATH (see Table 3 in Appendix). (Many tasks evaluated in self-consistency paper predate GPT3.5 and thus may have data contamination issue. The BBH data contains more recent and more challenging benchmark tasks, e.g. Date Understanding is considered more challenging than StrategyQA [3].) Results show that Reprompting still outperforms zero-shot and few-shot prompting consistently and substantially by 14-17 points on average. Compared with human-written CoT, Reprompting achieves better performance on all these tasks except on Date Understanding, Reasoning About Colored Objects, and MATH Algebra, on which the score differences are very small (<4 point). On average, Reprompting still outperforms CoT by +12.4 point on these tasks.

Reviewer comment (Experiments):
major: justification for the contribution "*Reprompting can transfer knowledge from a stronger model to a weaker model*" is not strong; only one model is studied, i.e., from GPT-3.5-turbo to text-davinci-003.

Author response:
We have downplayed the claim for the contribution "Reprompting can transfer knowledge from a stronger model to a weaker model" in the updated paper, as we observed it only on GPT3.5 to text-davinci-003. However, the finding is still interesting and serves as a starting point for further study in future works.

Reviewer comment (Experiments):
minor: no ablation study for the rejection probability (i.e., testing accuracy w.r.t. rejection probability).

Author response:
We further conducted an ablation study on the rejection sampling and recombination process. Results in Table 5 (in Appendix) show that, without rejection sampling, the test performance degrades greatly. Always rejecting solutions that lead to wrong answers also leads to degradation. Additionally, not allowing multiple solutions to be recombined when sampling new solutions at the iterative sampling stage also hurts performance.

Reviewer comment (Theory):
Is the rejection sampling used in Sec 2.2 equivalent to drawing samples according to (5)?

Author response:
Rejection sampling is used to simulate the sampling from Eq(5), but is not equivalent to it.

Reviewer comment (Evaluation):
"*Note that we set the rejection probability $p_{rej}$ in a way that allows solutions that lead to incorrect answers to be kept occasionally, as these solutions may still contain useful segments that evolve into good recipes through Reprompting*", any examples? How many incorrect answers are in the generated CoT receipts?

Author response:
We showed an example of incorrect solutions evolving into a correct one through iterations in Figure 3. The percentage of incorrect solutions kept in the initialization stage is in Figure 2 (at 0-th iteration), which is around 50-70%.

Reviewer comment (Experiments):
is reprompting sensitive to the choice of $\\{x_i, y_i\\}_{i=1}^N$?

Author response:
The performance does not vary much given different choices of the prompt examples after running Reprompting, which is probably because the solutions learned through Reprompting follow the same reasoning paths.

Reviewer comment (Experiments):
Will the proposed Reprompting be used in recent verification-based methods? e.g.  
  - Large Language Models are Better Reasoners with Self-Verification  
  - LEVER: Learning to Verify Language-to-Code Generation with Execution  
  - Forward-Backward Reasoning in Large Language Models for Mathematical Verification  
  - Making Language Models Better Reasoners with Step-Aware Verifier

Author response:
Yes, applying Reprompting to optimize the prompt for verification would indeed be a great future direction.

Reviewer comment (Theory):
the main reason for rejection is the assumptions required by the proposed algorithm are strong and difficult to check when applying the algorithm in practice.

Author response:
The assumptions in Eq(1)(2) are based on the empirical findings in [1,2] on a wide range of tasks. Specifically, [1] shows that the test performance does not vary much when increasing the number of supervised examples in the prompt from 8 to 32 or more. Chain-of-Thought (CoT) prompting further reduces the number of demonstrations needed -- the test performance remains similar when there is a varying number (but typically >= 3) of few-shot exemplars. These findings can be expressed formerly in mathematical forms as in Eq(1)(2), and have served as the base of LLM evaluation through few-shot prompting in prior work [3,4] . We have updated Section 2.1 to better explain it.

Reviewer comment (Theory):
the authors provided some references for the strong assumptions (1) and (2); however, it is still not convincing.

Author response:
The assumptions in Eq(1)(2) are based on the empirical findings in [1,2] on a wide range of tasks. Specifically, [1] shows that the test performance does not vary much when increasing the number of supervised examples in the prompt from 8 to 32 or more. Chain-of-Thought (CoT) prompting further reduces the number of demonstrations needed -- the test performance remains similar when there is a varying number (but typically >= 3) of few-shot exemplars. These findings can be expressed formerly in mathematical forms as in Eq(1)(2), and have served as the base of LLM evaluation through few-shot prompting in prior work [3,4] . We have updated Section 2.1 to better explain it.

Reviewer comment (Theory):
Therefore, users need to check whether the assumptions are satisfied before applying the proposed algorithm.

Author response:
The assumptions in Eq(1)(2) are based on the empirical findings in [1,2] on a wide range of tasks. Specifically, [1] shows that the test performance does not vary much when increasing the number of supervised examples in the prompt from 8 to 32 or more. Chain-of-Thought (CoT) prompting further reduces the number of demonstrations needed -- the test performance remains similar when there is a varying number (but typically >= 3) of few-shot exemplars. These findings can be expressed formerly in mathematical forms as in Eq(1)(2), and have served as the base of LLM evaluation through few-shot prompting in prior work [3,4] . We have updated Section 2.1 to better explain it.

Reviewer comment (Novelty):
Complex-CoT relies on a pool of human-written CoT recipes to select from while Reprompting does not — this is a valid motivation, but the reviewer thinks writing eight CoT demos is easy.

Author response:
We have conducted additional experiments to compare Reprompting with ComplexCoT (Table 4 in Appendix). On the three commonsense and arithmetic reasoning tasks, our method outperforms the baseline by +5.3 points on average. Note that Complex-CoT relies on a pool of human-written CoT recipes to select from while Reprompting does not. We are still running ComplexCoT+SelfConsistency (which is 8.5x more costly than Reprompting on GSM8K), but we don’t expect it to outperform Reprompting since SelfConsistency improves over CoT by <= 4 point in our previous experiment.

Reviewer comment (Experiments):
I am still a bit unconvinced that only five subtasks out of the 23 tasks of BIG-Bench Hard were considered, given that each subtask is quite small and choosing 5 out of 23 leads to doubts on whether the results are cherry-picked. In the previous reviewing round, the authors responded that they chose tasks where 1) CoT prompting is likely effective and 2) diverse, and they endeavoured to include additional tasks. However, I think in a practical setting, it's unlikely that practitioners will know beforehand whether CoT prompting is effective or not, and it is still interesting to see how methods such as Reprompting work *even when CoT prompting does not improve much*, as this gives readers stronger faith that in the worst case scenario, the method at least does not deteriorate performance.

Author response:
We thank the reviewer for the valuable comments and insightful questions.

Due to improved compute access recently, we were able to run more experiments on 10 additional tasks, including 7 tasks from BigBench Hard (BBH), GSM8K (math word problems) and two from Hendrycks’ MATH (see Table 3 in Appendix). On BBH, we intentionally selected the subtasks on which CoT does not improve much or does not improve consistently, such as Formal Fallacies, Movie Recommendation, Ruin Names, Salient Translation Error Detection, and Word Sorting. Results show that Reprompting still outperforms zero-shot and few-shot prompting consistently and substantially by 14-17 points on average. Compared with human-written CoT, Reprompting achieves better performance on all these tasks except on Date Understanding, Reasoning About Colored Objects, and MATH Algebra, on which the score differences are very small (<4 point). On average, Reprompting still outperforms CoT by +12.4 point on these tasks.

And interestingly, on tasks where CoT even underperforms zero-shot prompting, such as Movie Recommendation, Salient Translation Error Detection, and Word Sorting, Reprompting still improves over zero-shot prompting substantially. This suggests that not all CoT recipes improve model performance, and some may even lead to degradation. Reprompting serves as an effective algorithm for discovering and optimizing the CoT prompt to best exploit and compare LLMs.

Reviewer comment (Experiments):
Given the time allowance between the review cycles, I would expect experiments on additional subtasks, but this is not present in the manuscript. I see this as the largest room for improvement of the current manuscript.

Author response:
We thank the reviewer for the valuable comments and insightful questions.

Due to improved compute access recently, we were able to run more experiments on 10 additional tasks, including 7 tasks from BigBench Hard (BBH), GSM8K (math word problems) and two from Hendrycks’ MATH (see Table 3 in Appendix). On BBH, we intentionally selected the subtasks on which CoT does not improve much or does not improve consistently, such as Formal Fallacies, Movie Recommendation, Ruin Names, Salient Translation Error Detection, and Word Sorting. Results show that Reprompting still outperforms zero-shot and few-shot prompting consistently and substantially by 14-17 points on average. Compared with human-written CoT, Reprompting achieves better performance on all these tasks except on Date Understanding, Reasoning About Colored Objects, and MATH Algebra, on which the score differences are very small (<4 point). On average, Reprompting still outperforms CoT by +12.4 point on these tasks.

And interestingly, on tasks where CoT even underperforms zero-shot prompting, such as Movie Recommendation, Salient Translation Error Detection, and Word Sorting, Reprompting still improves over zero-shot prompting substantially. This suggests that not all CoT recipes improve model performance, and some may even lead to degradation. Reprompting serves as an effective algorithm for discovering and optimizing the CoT prompt to best exploit and compare LLMs.

Reviewer comment (Experiments):
Missing baseline comparison for parameter-efficient fine-tuning: since Reprompting is mainly pitched as a way to find high-performing prompts, it should also be compared against parameter-efficient methods such as soft-prompt tuning and LoRA. The comparison should include both absolute performance (Table 1) and convergence rate (Figure 2).

Author response:
Compared with prompt tuning and fine-tuning approaches, the number of labeled samples required by our approach is much less. Prompt tuning and parameter-efficient fine-tuning approaches require 100 to several thousand labeled samples for training and validation [1,2,3,4,5], which is not suitable for the few-shot setting that we focus on in this paper (e.g. on BigBenchHard, some tasks only contain a few dozen labeled samples apart from the 250 test samples). By contrast, our approach only requires 20 labeled samples, and thus can be tested on these few-shot benchmarks. However, we did add an additional baseline that optimizes the CoT prompt based on human-written solutions (so it’s human effort + optimization) [6]. The results are presented in Table 4 in the updated Appendix. On the three commonsense and arithmetic reasoning tasks, our method outperforms the baseline by +5 points on average.

Reviewer comment (Experiments):
Missing sampling ablation: at the top of page 4, the authors mention that the LLM likelihood is not always accessible so they approximate it by randomly rejecting new samples. This approximation needs more justification for why it’s appropriate. Additionally, there should be quantitative evidence supporting this approximation — specifically, how much better does the method work when actually computing the sampling probabilities? Even if probabilities are not available to many practitioners, it is important to know scientifically how much is lost with the approximation.

Author response:
Sampling ablation: Since the model likelihood is only available for text-davinci-003 (which is 10x more expensive than GPT3.5) but not GPT3.5, and sampling a set of candidates for reweighted sampling at each iteration is time-consuming, it takes much more time to run. But we are still trying to run it and incorporate the results in the updated version.

Reviewer comment (Experiments):
Missing rewriting ablation: line 14 of Algorithm 1 doesn’t condition on the current CoT \( z_j \), so the new one is sampled without reading prior guesses. It is important to investigate how much better Reprompting would work if it had access to prior CoT guesses. While this would diverge from the original Gibbs sampling motivation, it could provide substantial improvements, as suggested by Yang et al., 2023 (https://arxiv.org/abs/2309.03409).

Author response:
Reprompting+Rewriting: This would indeed be an interesting way to further improve the performance of Reprompting, although it remains an open question how to instruct a model so that they can use recipes in a prompt differently (i.e. it should try to recombine the reasoning paths provided by recipes on other questions while improving (but not copying) the previous recipe on the same question).

Reviewer comment (Experiments):
Limited benchmarks: since this paper rests entirely on experimental evaluations (with no theoretical contribution), the benchmarking should be more thorough. In principle, Reprompting could be applied to any task, so results on datasets such as Hendrycks Math, MMLU, and others were expected.

Author response:
We have added more experiments on 10 additional tasks and popular benchmarks, including 7 tasks from BigBench Hard (BBH), GSM8K (math word problems) and two from Hendrycks’ MATH (see Table 3 in Appendix). Results show that Reprompting still outperforms zero-shot and few-shot prompting consistently and substantially by 14-17 points on average. Compared with human-written CoT, Reprompting achieves better performance on all these tasks except on Date Understanding, Reasoning About Colored Objects, and MATH Algebra, on which the score differences are very small (<4 point). On average, Reprompting still outperforms CoT by +12.4 point on these tasks.

Reviewer comment (Writing):
Exposition issue: the method feels as if it has been forced to fit the Gibbs sampling narrative, with many approximations introduced along the way. It is unclear why this framing was chosen, especially since the authors do not use the Gibbs motivation later on. Additionally, the evolutionary algorithm view is briefly mentioned before Section 3, but its purpose is not explained.

Author response:
Exposition: The motivation for finding effective CoT recipes using Gibbs sampling is that an effective CoT recipe for solving a set of problems should be the one that can be applied to generating accurate solutions for all these problems, i.e. a common reasoning path. The distribution of such recipes is unknown, but we know how the solutions to these problems should be correlated. Thus, we can use Gibbs sampling to address the problem. Empirically, we found that the CoT solutions produced by Reprompting typically end up following a similar format and reasoning path, which verifies that by applying Gibbs sampling along with other approximations, we are able to discover common reasoning path for solving a set of problems.

Reviewer comment (Experiments):
How does Reprompting compare to parameter-efficient tuning methods?

Author response:
Compared with prompt tuning and fine-tuning approaches, the number of labeled samples required by our approach is much less. Prompt tuning and parameter-efficient fine-tuning approaches require 100 to several thousand labeled samples for training and validation [1,2,3,4,5], which is not suitable for the few-shot setting that we focus on in this paper (e.g. on BigBenchHard, some tasks only contain a few dozen labeled samples apart from the 250 test samples). By contrast, our approach only requires 20 labeled samples, and thus can be tested on these few-shot benchmarks. However, we did add an additional baseline that optimizes the CoT prompt based on human-written solutions (so it’s human effort + optimization) [6]. The results are presented in Table 4 in the updated Appendix. On the three commonsense and arithmetic reasoning tasks, our method outperforms the baseline by +5 points on average.

Reviewer comment (Theory):
Can you provide a justification for the approximation at the top of page 4?

Author response:
Sampling ablation: Since the model likelihood is only available for text-davinci-003 (which is 10x more expensive than GPT3.5) but not GPT3.5, and sampling a set of candidates for reweighted sampling at each iteration is time-consuming, it takes much more time to run. But we are still trying to run it and incorporate the results in the updated version.

Reviewer comment (Evaluation):
How much performance is lost due to this approximation? Is it problematic that generative LLMs are bad at estimating the likelihood of their samples, and how does that affect the Gibbs sampling motivation?

Author response:
Sampling ablation: Since the model likelihood is only available for text-davinci-003 (which is 10x more expensive than GPT3.5) but not GPT3.5, and sampling a set of candidates for reweighted sampling at each iteration is time-consuming, it takes much more time to run. But we are still trying to run it and incorporate the results in the updated version.

Reviewer comment (Evaluation):
In terms of wall-clock time, how long did it take to converge in Figure 2?

Author response:
The wall-clock time for running Reprompting largely depends on the rate limit and delay of API calls to these LLMs. In our experiment, it takes 4-10 hours to run Reprompting on a single task.

Reviewer comment (Evaluation):
The motivation for decoupling initialization and sampling is to improve the diversity of initial prompts. Is there evidence for this claim? From Table 1, it is unclear when to use one LLM for both phases versus decoupling them.

Author response:
The motivation for decoupling initialization and sampling is based on the observation that some LLMs (e.g. GPT3.5) are better at generating more diverse reasoning paths than some others (e.g. text-davinci-003). Thus, using the former ones as the initialization model leads to a higher chance of discovering better CoT recipes for a task.

Reviewer comment (Evaluation):
Did you observe convergence in the type of CoT prompts? Do they all end up following a similar format, and is that problematic for some tasks?

Author response:
The CoT prompts produced by Reprompting typically end up following a similar format and reasoning path. This is beneficial for finding a common, effective reasoning path for a specific type of problem. But for problems where the reasoning paths may vary greatly, it is best to first cluster the problems based on their type (e.g. by prompting LLMs about the category of the problem, whether they belong to dynamic programming, optimization, graph search or other problem types) or the clustering of their zero-shot solutions.

Reviewer comment (Experiments):
There is no ablation study. It is not clear which elements of the algorithm are most important for its success.

Author response:
We further conducted an ablation study on the rejection sampling and recombination process. Results in Table 5 (in Appendix) show that, without rejection sampling, the test performance degrades greatly. Always rejecting solutions that lead to incorrect answers also leads to degradation. Additionally, not allowing multiple solutions to be recombined when sampling new solutions at the iterative sampling stage also hurts performance. Furthermore, since the Auto-CoT baseline in our experiment differs from our method in that it only has the initial sampling step without the iterative sampling step, it also serves as an ablative baseline. The comparison results show that the iterative sampling step brings +11 point improvement on average on the five BBH tasks.

Reviewer comment (Experiments):
I do not feel comfortable accepting this paper until there is evaluation against prompt optimization schemes at a similar compute budget. Currently, by simply comparing against baselines it is too hard to gauge the contribution of this paper.

Author response:
Compared with prompt tuning and fine-tuning approaches, the number of labeled samples required by our approach is much less. Prompt tuning and parameter-efficient fine-tuning approaches require 100 to several thousand labeled samples for training and validation [1,2,3,4,5], which is not suitable for the few-shot setting that we focus on in this paper (e.g. on BigBenchHard, some tasks only contain a few dozen labeled samples apart from the 250 test samples). By contrast, our approach only requires 20 labeled samples, and thus can be tested on these few-shot benchmarks. However, we did add an additional baseline that optimizes the CoT prompt based on human-written solutions (so it’s human effort + optimization) [7]. The results are presented in Table 4 in the updated Appendix. On the three commonsense and arithmetic reasoning tasks, our method outperforms the baseline by +5 points on average. 
	The fact that optimized prompts can beat human-written ones on reasoning tasks is not unsurprising. For instance, a recently proposed approach on optimizing/adapting CoT prompts does not outperform human-written CoT on average [6]. However, our method does outperform human-written CoT by large margins on a wide range of reasoning tasks (see Table 1 and the additional results on other tasks in Table 3 in Appendix).

Reviewer comment (Theory):
Some more justification on the approximation of the equation 4) needs to be provided. Specifically, the eq 4) suggests some form of conditional independence structure in the fully connected graphical model, but how well it approximates (or how the neighborhood size matters) needs to be justified.

Author response:
In Eq(4), the first proportionality is deduced based on Bayes’ Theorem. The second approximation is based on the in-context learning assumption discussed in Eq(2). This assumption is based on the empirical findings in [3,4] on a wide range of tasks that the test performance remains similar when there is a varying number (but typically >= 5) of few-shot exemplars. We have updated Section 2.1 to better explain it.

Reviewer comment (Experiments):
The few-shot experiment setting might need a bit more justification. For example if the authors used 20 examples during CoT generation, then it might be tricky to call it a K-shot learning when K examples are used during the inference stage.

Author response:
Our method uses 20 examples for CoT prompt generation, and the final prompt contains 5 example CoT. Since 20 labeled samples are used, it satisfies the definition of 20-shot learning. This is slightly different from the K-shot learning in standard few-shot prompting where K is the number of examples in the final prompt. But our comparison with few-shot prompting is still fair, since we are comparing our method with 20-shot prompting. As for the CoT baseline, since the number of CoT demonstrations is limited (and often less than 20) and each individual demonstration takes more tokens, we just take all the available CoT demonstrations that could fit into the prompt under the token limit.

Reviewer comment (Experiments):
Some latest baselines regarding the automatic prompt engineering need to be discussed or compared if appropriate, for example [1][2].

Author response:
[1] proposed a novel self-adaptive prompting approach to improve model performance on reasoning tasks in zero-shot settings. In their approach, they generate step-by-step solutions through zero-shot CoT prompting and select demonstration examples from the zero-shot solutions via carefully designed criteria. This is different from our approach in that our approach is iterative so that zero-shot solutions can be recombined (Figure 4) and evolve into better recipes (Figure 3) through iterations. Empirically, since the approach in [1] does not outperform standard CoT (one of our baselines) on average, and their code has not been released yet, we didn’t include [1] as a baseline. But we will incorporate the discussion in paper. [2] proposed a novel prompt tuning method via reinforcement learning. The approach is based on a set of prompt editing operations that are designed for text classification tasks. Such approach is not directly applicable to reasoning tasks where the space of all possible step-by-step reasoning paths is huge, and one needs to explore the space by discovering and recombining new paths for solving the problem rather than simply rephrasing existing ones. We will incorporate the discussion and citations in the paper.