Paper: SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning (ICLR-2024)

Reviewer comment (Experiments):
SelfCheck requires 3 LLM queries per substep. Consequently, the comparison between SelfCheck and SelfConsistency appears skewed, as SelfCheck inherently utilizes approximately 10x more samples (assuming an average of 3 substeps). Furthermore, the results with GPT-4 are based on a limited sample size, and given the analysis with Self-Consistency, the findings should be interpreted cautiously.

Author response:
Thanks for the great question! We have a discussion on the cost of SelfCheck at the end of the general response. The cost of SelfCheck is at most 2x the cost generator on GPT-3.5 (as the cost is calculated based on tokens rather than api calls) and we think that the benefits of using SelfCheck will often justify its cost. For example, Figure 5 shows that majority voting can quickly saturate beyond a small number of solutions while SelfCheck does not, while the new Appendix D shows there are cases where majority voting cannot be used but SelfCheck can.

Moreover, as discussed at the start of the general response, the utility of SelfCheck extends beyond just improving final predictive accuracy.  As you say, showing that LLMs can check their answers and rectify their own biases is of significant finding in itself, with various practical benefits of its own.

Reviewer comment (Experiments):
While the method is presented as 0-shot, it requires training samples (as seen in section 3.2) to learn hyperparameters, and thus, the criticism of approaches that require examples is unfair (We note though that these approaches are not directly comparable to SelfCheck in general, as they require additional exemplars which will often not be available in practice.). Additionally, the authors had to meticulously craft instructions for each step, which somewhat undermines their critique of other methods.

Author response:
SelfCheck is a real zero-shot method, and the hyper-parameters $\lambda_{-1}$ and $\lambda_{0}$ are fixed rather than learned from training samples: they are constant across all experiments and were chosen based on a small amount of upfront testing, rather than full hyperparameter tuning.  In particular, this initial testing suggested any reasonable values for $lamda_{-1}$ that are significantly higher than $lamda_0$ worked, so we never attempted to learn or directly tune them in practice.

Our instructions are very generic and kept the same across experiments, rather than being customized to different problems. We believe this significantly distinguishes it from other work, where the examples used are generally problem-specific.

When designing these, we used only 5 examples from MathQA and simply checked whether GPT-3.5 followed the instructions in our prompts and slightly adjusted the prompts if not. We did not do any direct tuning of the instructions beyond this.

Reviewer comment (Experiments):
If my observations in Weakness A are accurate, could Figure 5 simply result from comparing incompatible data points? Should we compare SelfCheck @ k with SelfConsistency @ m with m ~ 10 x k?

Author response:
That’s a great question! We continued running the majority voting baseline on MathQA until the solution number reached 200. However, the observed saturation of majority voting continues, with accuracy continuing to vibrate around 0.78~0.79, which is the accuracy achieved by SelfCheck with 5 solutions and about 0.07 points lower than SelfCheck when the number of solutions is 30. As the cost of each SelfCheck verification step is about double that of a generation step here, it means that it is outperforming in a cost-normalized manner once it uses 5 or more solutions.  

We would also again like to reiterate here that majority voting cannot always be used (e.g. the new experiment in Appendix D) and that SelfCheck provides benefits beyond final predictive performance.

Reviewer comment (Evaluation):
What are the false positive rates at various thresholds?

Author response:
The false positive rates at various thresholds are:

|&nbsp; &nbsp;&nbsp; $t$|0.0|0.5|0.6|0.7|0.8|0.9|0.93|0.96|0.99|1.0|
| :- | :-: |:-: |:-: |:-: |:-: |:-: |:-: |:-: |:-: |:-: |
|GSM8K|1.0|0.98|0.94|0.87|0.71|0.49|0.31|0.31|0.31|0.0|
|MathQA|1.0|0.99|0.96|0.89|0.70|0.38|0.20|0.08|0.08|0.0|
|MATH|1.0|0.98|0.94|0.87|0.73|0.45|0.36|0.18|0.18|0.0|

Reviewer comment (Experiments):
When devising the prompts, a small number of training samples from MathQA dataset were utilized. How were these examples used?

Author response:
Our instructions are very generic and kept the same across experiments, rather than being customized to different problems. We believe this significantly distinguishes it from other work, where the examples used are generally problem-specific.

When designing these, we used only 5 examples from MathQA and simply checked whether GPT-3.5 followed the instructions in our prompts and slightly adjusted the prompts if not. We did not do any direct tuning of the instructions beyond this.

Reviewer comment (Experiments):
I think discussing how to prompt some "incantations" to the LLMs (especially for such black-box LLMs) has no insight into this community. The proposed prompts for each component in the self-checking process are well-designed, especially for GPT-serial LLMs. In my opinion, they could not be well extended to other LLMs, such as LLama or Claude.

Author response:
We have added an extra experiment using  Llama2 (70b) on GSM8K, which shows that SelfCheck generalizes well: we actually see a bigger gain from using SelfCheck than for GPT-3.5.
Please see Q2 in the general response and the new Appendix E for more details. We unfortunately do not have access to Claude at the moment so cannot test it.  We find that the accuracy of the generator when using Llama2 is too poor to provide any meaningful results on MathQA and MATH (as almost all generations are incorrect).  

More generally, we strongly disagree that the contribution of our work is in prompting “incantations”.  The key aspect of SelfCheck is in its novel decomposition of the step verification problem, with the precise prompts used very generic and not finely tuned to the particular setup (e.g. the prompts are held fixed across datasets).  Showing that LLMs are capable of checking their responses is also of significant scientific interest in its own right.


Finally, the vast majority of  LLM works involve prompt designing, including the very famous Chain-of-Thought~(CoT) paper [1], which is purely a prompt designing work but has significantly influenced the community.  We thus do not feel this would be reasonable grounds for rejecting the paper, even if it were a fair characterization.

[1] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al.

Reviewer comment (Theory):
Step-by-step checking may have the problem of error propagation.

Author response:
We feel there may have been a significant misunderstanding here: though error propagation can indeed be an issue with CoT reasoning itself, SelfCheck is itself a mechanism to catch such errors and does not suffer from propagation of errors in its own verifications. Note here that SelfCheck verifies each reasoning step *independently*, conditional on information gathered from its context. If an error in our verification happens during verification of one step, it won’t affect other verification steps as the results of earlier verifications are not used in the verification process of later steps.

Reviewer comment (Theory):
Why the authors do claim the solution is self-checking? In Table 1, the authors also employ different LLMs on the Generator and Checker.

Author response:
We focus on the key idea that an LLM can check its own reasoning because we believe that this is of scientific interest in itself, while most of our experiments are using the same LLMs as the generator and the checker. The cross-checking ability is an extension to this self-verification ability which might often be of additional practical use, but does not detract from our main contribution.

Reviewer comment (Experiments):
Are the proposed checking components and prompts in each step useful for other LLMs?

Author response:
Yes. The prompts were designed with GPT-3.5 on 5 examples from MathQA, but were used in every experiment in the paper. Transfer has already been shown between GPT 3.5 and GPT 4, as well as different datasets. As previously mentioned, we have also done an extra experiment to verify the generalization ability of SelfCheck to Llama2 and found strong performance.

Reviewer comment (Presentation):
Why there is no dot for DV and SV in Figure 2 (b)

Author response:
The plotted values are taken directly from the reported results in the respective DV and SV papers, neither of which provided results for MathQA.  Unfortunately, due to budget restrictions, we were unable to rerun their approaches to provide our own values (and thus provide results for them on MathQA).

Reviewer comment (Presentation):
What is the meaning of the yellow curve in Figure 3?

Author response:
It's the proportion of real wrong solutions in solutions that are predicted to be wrong. We have made it clearer in the paper.

Reviewer comment (Experiments):
I think that the discussion in subsection 5.2 is of no use. The performance heavily relies on the prompts. So I think if we adopt an optimal prompt in the global checking setting, better results may be obtained.

Author response:
We believe that our results directly provide strong evidence against this assessment.  This section is an ablation to try and establish if a global checker can achieve similar performance with the right prompt, and we emphatically find that this is not the case. Note that we carefully tried to tune this global prompt to try and make it effective (this is actually how we started the project and the idea of the SelfCheck approach came out of the fact that we could not make it work). Others have also similarly presented results suggesting this does not work, see e.g. Table 2 in [2]. 


[2] Deductive verification of chain-of-thought reasoning. Ling et al.

Reviewer comment (Experiments):
Why were the experiments conducted on the math-related tasks? I wonder where the performance goes when other types of reasoning are explored, such as logical reasoning, deductive reasoning, and temporal reasoning.

Author response:
This is a fair question and we agree that experiments on other tasks will improve the paper.  To this end, we have performed an extra experiment on logic reasoning using the SNR dataset. Please find the results in the general response, Q1, and the new Appendix D in the updated paper. We find that SelfCheck again provides benefits in this setting.

We note here that the original focus on math-based datasets is in line with most other works in the literature, and was primarily based on dataset availability and ease of performance evaluation.  For example, our new experiment required volunteers to manually evaluate the answers.

Reviewer comment (Experiments):
The authors claim that they provide a general-purpose, zero-shot approach to checking. However, the proposed schema is more tailored to maths problems to some extent as answering mathematical questions requires explicit computational steps and numerical formulas, which can be easily checked by LLMs. Hence, it is a little confusing to state the proposed SelfCheck is general-purpose. I wonder if the authors could conduct experiments on other types of reasoning tasks such as logical reasoning or commonsense reasoning.

Author response:
This is a great suggestion that we think will improve the paper. We’ve added an experiment on logic reasoning with the dataset of SNR, which shows the generalization ability of SelfCheck to non-math reasoning problems. We find that SelfCheck is effective in this setting as well. Please see Q1 in the general response and the new Appendix D for details.

Reviewer comment (Evaluation):
The authors check each step by evaluating the conditional correctness of each step based on the provided question and previous steps in the solution. I doubt whether there could be a phenomenon where step i is wrong, but the following steps are perfectly correct, i.e. the model is not affected by the step i error. But if the authors evaluate using conditional errors, it’s quite possible that the model would determine all subsequent steps (after step i) are wrong. Could the authors provide more analysis or explanations?

Author response:
Thank you for raising this, we may have not made it sufficiently clear what we mean by the 'conditional correctness' of a step previously. The evaluation of each step is evaluated independently and does not depend on whether previous errors were detected: we assess correctness conditioned on taking the information gathered from context to be true.  We find many cases where subsequent steps are conditionally correct extensions of previous erroneous steps. They do not have to reach a correct final conclusion for conditional correctness to hold; indeed miraculous corrections back to the true correct answer are much rarer.

To give a concrete example, if we had the question “What is 4x(3+2)?” and Step 1 concluded that 3+2=6, then it would be conditionally correct for Step 2 to conclude that 4x(3+2)=24, conditional on the information in context that 3+2=6.

We have clarified this in the updated submission.

Reviewer comment (Experiments):
The design of the integration function seems helpful as shown in the experimental results. Could the authors explain the insights of this design and the choices of hyperparameters?

Author response:
In order not to introduce extra complexity, the integration function is simply a linear model. 
The hyper-parameters $\lambda_{-1}$ and $\lambda_{0}$ are fixed across all experiments and were chosen based on a small amount of upfront testing, rather than full hyperparameter tuning. The introduction of the hyper-parameters $\lambda_{-1}$ and $\lambda_{0}$ is to instill the inductive bias that 'contradict' steps have more effect on the overall correctness than 'not related' steps into SelfCheck. In particular, this initial testing suggested any reasonable values for $lamda_{-1}$ that are significantly higher than $lamda_{0}$ worked, so we never attempted to learn or directly tune them in practice.

Reviewer comment (Evaluation):
The authors claim that SelfCheck significantly outperforms majority voting with both GPT-3.5 and GPT-4. However, results in Table 1 (Row 2) show that SelfCheck exhibits inferior performance in GSM8K with GPT-4 as the generator and checker.

Author response:
The results are very close and not statistically significant in this case. However, it is not immediately clear why SelfCheck is not helpful in this single configuration. We guess it might be because GPT-4 is already strong enough to solve the simple problems in GSM8K without being affected by its biases. We note that gains are significant in almost every other setting.

Reviewer comment (Experiments):
Results in Figure 2 demonstrate that the SelfCheck did not exceed majority voting by much, but the inference cost of SelfCheck should be larger in comparison. Could the authors illustrate in detail the superiority of SelfCheck over majority voting?

Author response:
We would first like to highlight here, as discussed at the start of the general response, that SelfCheck is more than a method for improving predictive accuracy. It is thus immediately beneficial over majority voting in showing the crucial insight that LLMs are capable of checking their own answers in a meaningful way and allowing tasks such as verification, confidence scoring, and filtering of individual answers.

Second, we believe that there are important ways that SelfCheck can be better than majority voting in a larger ensemble from a purely predictive perspective:
- As shown in Figure 5, the performance of majority voting saturates with the number of solutions.  It never does better than SelfCheck with 5 solutions, no matter how many solutions are used, with SelfCheck giving around a 7\% gain in performance for large ensembles.
- There are many cases in which majority voting cannot be used because answers are in free text form rather than being numeric, see the new experiment in Appendix D.

Finally, we would also like to point out that the performance gap between SelfCheck and majority voting is significant. Selfcheck increases accuracy by ~1/% to ~4\% compared with majority voting for all settings. By comparison, previous verification-based methods such as Deductive Verification and Self-Verification only provide an increase of at most 1\%.

Reviewer comment (Writing):
The writing can be improved. There are some typos and unclear descriptions. Please refer to the questions below for details.

Author response:
Thank you for pointing them out; we have corrected them.

Reviewer comment (Writing):
In Paragraph 2, Section 3.1, the authors state that ‘checking’ is a less common task in the training corpus of most LLMs. Can the authors provide some evidence of citations of this statement?

Author response:
Sorry this wasn't so clear in the paper. We are simply alluding to the fact that they are trained as generative models, instead of directly trained as methods for checking. The comparison in Section 5.2 shows the superiority of regeneration & comparison compared with direct error checking, which supports the argument that they are more suited to this generation task. We have made edits to make this clear.

Reviewer comment (Experiments):
In Paragraph 5, Section 3.1, the authors claim that LLM is usually able to perform the tasks of figuring out the target of the current step. Missing evidence or supplementary experiments should be added.

Author response:
We find that SelfCheck can extract step targets very accurately in practice relative to the regularity of errors in the checking itself. It’s difficult to directly evaluate this quantitatively, as there is no absolute measure of its correctness for the target extraction and even trying to do our own manual labelling becomes quite subjective.  However, we have added some randomly selected examples in Appendix C to show its typical behaviour.

Reviewer comment (Writing):
In Section 3.1, the authors mention Information in both stages of Information collection and Step regeneration but the Information should refer to different contents according to the authors’ descriptions. A minor suggestion would be to modify the terms in the prompts in order to better distinguish them.

Author response:
We use the same terms as the “Informations” in step regeneration are always a renumbered subset of those in information collection.  Note here $I_0$, $I_1$, ... are being directly used to index from the original set in the information Collection stage. For example, if $I_0=2$ in step regeneration, Information 2 from the question is used, and renumbered as the 0-th information to be fed into the regeneration stage.

Reviewer comment (Presentation):
The comparison between the proposed SelfCheck and DV/SV in Figure 2 is quite confusing. Could the authors give relevant explanations on how they compare these methods with varying scores and different generator models?

Author response:
Because of budget constraints, it is not possible for us to repeat DV and SV by ourselves. Instead, we compare gains in accuracy over majority voting based on their reported results (noting that differences in generators mean that absolute accuracies are not comparable), matching the number of solutions used by each paper (10 for DV and 5 for SV). In both cases that we also test DV is actually worse than performing no checking, while SV has a gain of less than 0.5\%.  More generally, both approaches only very rarely provided gains of more than 1\% over majority voting across their experiments. Thus the gains we have demonstrated for SelfCheck are noticeably bigger than those reported in any setting by those papers.

Reviewer comment (Experiments):
Why did not the authors conduct experiments with GPT-3.5 as the Generator and GPT-4 as the Checker (in Table 1)? It would be helpful to provide appropriate descriptions.

Author response:
The reason is that using GPT-4 to check GPT3.5 is not a reasonable setting in practice and our limited budget meant we needed to be selective on what was run. The default setting of SelfCheck is to use the same kind of LLMs to check themselves. We further added the potentially useful case of using a cheaper model (GPT-3.5) to check more powerful (and expensive) ones (GPT-4). Using a more powerful model to check a cheaper one doesn't make much sense, because directly using the more powerful model to generate solutions would usually result in better performance.

Reviewer comment (Writing):
Minor comments on writing:  
(1) Paragraph 2 in Section 1: ..., which is far below human level -> , which is far below the human level  
(2) Paragraph 5 in Section 1 & Paragraph 1 in Section 3: to check their working -> check their work  
(3) Paragraph 1 in Section 3.2: , and thus final solution -> , and thus the final solution

Author response:
Thank you for pointing them out; we have corrected them.

Reviewer comment (Theory):
The paper presents a general purpose validation framework designed to assess the accuracy of multi-step mathematical reasoning tasks. Although this approach performs effectively well within a zero-shot prompting setup, it essentially relies on well-crafted prompts. It remains uncertain whether these four steps constitute the exclusive and comprehensive way of rigorously assessing the accuracy and the reasoning process in such tasks. There might be more ways to assess the quality of the reasoning chain and there is no guarantee that this is the only way or the best way. These four steps are not acquired through learning, generated by a model, nor rooted in learning science or based on any other theory. So it is very hard to judge it. It is plausible that their effectiveness is coincidental in the context of mathematical reasoning tasks and may not readily apply to other types of reasoning. Consequently, for broader applicability, the prompts would need to be rethought and rephrased, raising questions about the approach's generalizability.

Author response:
> (This approach) essentially relies on well-crafted prompts… It is plausible that their effectiveness is coincidental in the context of mathematical reasoning tasks and may not readily apply to other types of reasoning.

To address your concern, we have added an extra experiment on logic reasoning, which shows that SelfCheck generalizes well to non-mathematical reasoning tasks. Please see the response to Q1 in the general response and the new Appendix D in the updated paper.

&nbsp;

> It remains uncertain whether these four steps (in SelfCheck) constitute the exclusive and comprehensive way of rigorously assessing the accuracy and the reasoning process in such tasks. There might be more ways to asses the quality of the reasoning chain and there is no gurantee that this is the only way or the best way.
 
SelfCheck shows that LLMs can effectively detect their own errors, which answers an important question in the LLM community and is an important contribution in its own right. We agree that the idea could potentially be adapted and built on further, but this does not mean it is not already useful. We actually view this as a strength of the work as we are opening up new avenues for research and feel it would be very counterproductive to reject the paper on this basis. Critically, we are not aware of any previous work that has achieved a comparable level of performance in checking LLM CoT reasoning (especially not in a zero-shot manner).

Reviewer comment (Experiments):
The paper lacks a comparative analysis with other methods that have employed extensive prompts, such as PHP, faithful decomposition, verification based on subquestion decomposition, and similar approaches. So it is hard to judge if this prompting strategy is the best one for mathematical reasoning.

Author response:
> The paper lacks a comparative analysis with other methods that have employed extensive prompts, such as PHP , faithful decomposition, verification based on subquestion decomposition, and similar approaches. So it is hard to judge if this prompting strategy is the best one for mathematical reasoning.

Verification is itself an important question, so step-by-step verification of SelfCheck is useful in its own right. The aim of SelfCheck is checking and verification, which can help with downstream prediction, but this is not a paper rooted in simply trying to maximize final empirical predictive performance. 

We don’t compare to methods like PHP and faithful decomposition as they are aimed to directly increase predictive performance, but they are not capable of checking their own (and others) answers in a useful way. Moreover, they are not directly competing methods: SelfCheck can be used on top of them to check for their errors and further improve their performance.

Reviewer comment (Evaluation):
No comparison has been drawn to the cost of the prompting method, as for each sample, 4 extra verification steps are needed which makes 5 API calls (at least) per sample. Compared to one-step verification tasks that take 2 API calls, it is unclear whether the cost vs accuracy tradeoff is worth it.

Author response:
> No comparison has been drawn to the cost of the prompting method as for each sample, 4 extra verification steps are needed which makes 5 API calls (at least) per sample. Compared to one step verification tasks that take 2 API calls, is the cost vs accuracy tradeoff worth it?

Please see the response to Q3 in the general response. In short, we believe there are many settings where the benefits of using SelfCheck justify its cost. For example, Figure 5 shows that majority voting can quickly saturate beyond a small number of solutions while SelfCheck does not; the new Appendix D shows there are cases where majority voting cannot be used but SelfCheck can; SelfCheck can be used for verifying and filtering answers, rather than just to maximize predictive performance; and it can be used to identify the exact errors made in a CoT reasoning in an interpretable way.

Reviewer comment (Experiments):
Table 1 is a bit confusing as 2 samples for majority voting does not make much sense. There should be at least 3 so that there are some agreements between the samples.

Author response:
> Table 1 is a bit confusing as 2 samples for majority voting does not make much sense. Should be at least 3 so that there are some agreements between the samples.

We agree that MV has a failure mechanism when only using 2 samples and so is essentially the same as the default prediction in this setting. We have made this clearer in the paper, highlighting that Figure 2 (and the new Tables D.1 and E.1) should be used as the main basis of comparison between SelfCheck and MV.  

What we want to show in Table 1 is that SelfCheck can pick out the right answer when different runs of the generator don’t agree with each other. For example, when the generator and checker are both GPT-3.5, SelfCheck has 12.2%, 20.7%, 9.1% higher probabilities to pick out the right answers for each dataset compared with random selection. This can also be useful in cases where the generator is no longer available, so we need to select the best solution from the solutions already generated.