Paper: Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning (ICLR-2024)

Reviewer comment (Novelty):
A related paper with a similar idea, called "Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning with LLMs" (https://arxiv.org/pdf/2305.11860.pdf), was not referenced.

Author response:
Thanks for reminding and we will add citations to [1] and add more discussions on the novelty of our method:

(1) Having additional control scheme

Besides basic early-stopping self-consistency (ESC), we also propose its control scheme to dynamically choose the performance-cost balance. As we all know, along with the number of samples increases, performance will increase as well. For the realistic requirements, it is more practical to choose the affordable sample size rather than just try to maximize performance. We have derived the expectation of $\hat{L}$ (sampling number) and the upper bound of $\mathbb{E}(Q)$ (inconsistent probability). The experiments in Section 3.3 have shown that the predictions we obtain based on Equation 10 and Equation 14 are highly reliable for balancing sampling cost and voting performance.

(2) Less sampling cost

Adaptive-consistency (AC) generates samples step by step, which means each sample requires one input. Considering the demonstrations for in-context learning (usually 8 examples) have a lot of tokens, it will cost quite a portion of the budget. By contrast, ESC generates samples in multiple sampling windows, thus samples within one window can share the same input. Table 7 in updated paper shows that ESC can get higher accuracy with less sampling cost comparing with AC. A quick preview is below:

| Method      |    \# prompt | \# completion  | Cost | Acc|
| :--------: | :--------:| :--: | :--: | :--: |
| SC  | 496.9 |  2909.0   | 0.0084 | 85.69|
| AC     |   4930.3 |  813.2  | 0.0087|85.66|
| ESC      |    1469.2 | 1220.8  | 0.0052|85.67|

(3) No hyperparameters

It is necessary for AC to tune the hyperparameter, specifically the confidence threshold. But for proposed ESC, the stopping criterion needs no hyperparameter due to the most conservative strategy to maintain the performance, i.e., all the answers within a window are same. Thus ESC can be conducted directly for different tasks and models, without any validation set. Besides, section 3.2 has proved the robustness regrading to window size w and max sampling size L.

(4) More solid theoretical guarantees 

We provide more solid theoretical guarantees than AC lying in two parts: firstly, we rigorously derive the theoretical upper bound of the probability being inconsistent with the case where sampling size is infinite (true distribution of the model predictions according to Equation 3). While AC only considers the condition with next n samples, which is less rigorous; secondly, the upper bound of ESC is much more tight than AC. According to Equation 8, the bound value is only in the interval of $\leq 4 \times 10^{-2}$ to $\leq 2 \times 10^{-5}$ when window size varies from 3 to 10. By comparison，the confidence threshold of AC is chosen as 95\%.

Reviewer comment (Evaluation):
In the "SQA" dataset, Lˆ-SC outperforms both SC and ESC, even though SC has a larger sample size. If there are no data errors, is there any possible reasonable explanation?

Author response:
We apologize that we have filled in the inconsistent values about SQA. That is because the candidate answer for question in SQA is either 'True' or 'False', but there will be some cases that the model generates neither of them. So there are two ways to vote the final answer: one is regarding these noisy answers as another type, e.g. '-1', to be included for voting; the other is removing this type of answers and only voting 'True' or 'False'. Our first choice was to vote based on the former criterion, but later we realized that the latter one made more sense, but we have forgotten to correct some values. We have fixed this problem in the revision.

Reviewer comment (Novelty):
The novelty of this work is unclear. [1] already proposed an adaptation of self-consistency to reduce the sampling cost, but this work did not cite and discuss this prior work. Without a thorough discussion and direct comparison, it is unclear whether ESC is more effective.

Author response:
**Comparison with Adaptive-Consistency and the novelty of our work**

Thanks for reminding and we will add citations to [1] and add more discussions on the novelty of our method:

(1) Having additional control scheme

Besides basic early-stopping self-consistency (ESC), we also propose its control scheme to dynamically choose the performance-cost balance. As we all know, along with the number of samples increases, performance will increase as well. For the realistic requirements, it is more practical to choose the affordable sample size rather than just try to maximize performance. We have derived the expectation of $\hat{L}$ (sampling number) and the upper bound of $\mathbb{E}(Q)$ (inconsistent probability). The experiments in Section 3.3 have shown that the predictions we obtain based on Equation 10 and Equation 14 are highly reliable for balancing sampling cost and voting performance.

(2) Less sampling cost

Adaptive-consistency (AC) generates samples step by step, which means each sample requires one input. Considering the demonstrations for in-context learning (usually 8 examples) have a lot of tokens, it will cost quite a portion of the budget. By contrast, ESC generates samples in multiple sampling windows, thus samples within one window can share the same input. Table 7 in updated paper shows that ESC can get higher accuracy with less sampling cost comparing with AC. A quick preview is below:

| Method      |    \# prompt | \# completion  | Cost | Acc|
| :--------: | :--------:| :--: | :--: | :--: |
| SC  | 496.9 |  2909.0   | 0.0084 | 85.69|
| AC     |   4930.3 |  813.2  | 0.0087|85.66|
| ESC      |    1469.2 | 1220.8  | 0.0052|85.67|

(3) No hyperparameters

It is necessary for AC to tune the hyperparameter, specifically the confidence threshold. But for proposed ESC, the stopping criterion needs no hyperparameter due to the most conservative strategy to maintain the performance, i.e., all the answers within a window are same. Thus ESC can be conducted directly for different tasks and models, without any validation set. Besides, section 3.2 has proved the robustness regrading to window size w and max sampling size L.

(4) More solid theoretical guarantees 

We provide more solid theoretical guarantees than AC lying in two parts: firstly, we rigorously derive the theoretical upper bound of the probability being inconsistent with the case where sampling size is infinite (true distribution of the model predictions according to Equation 3). While AC only considers the condition with next n samples, which is less rigorous; secondly, the upper bound of ESC is much more tight than AC. According to Equation 8, the bound value is only in the interval of $\leq 4 \times 10^{-2}$ to $\leq 2 \times 10^{-5}$ when window size varies from 3 to 10. By comparison，the confidence threshold of AC is chosen as 95\%.

Reviewer comment (Experiments):
In Table 1, when comparing ESC and L-SC, the performance difference is generally small. The reason can be that the improvement of SC saturates when the sample size increases, thus reducing the sampling size also does not drastically degrade the performance for SC. It is helpful to show this comparison for smaller sampling sizes, e.g., those in Table 2, and see if the performance improvement achieved by ESC can be more significant.

Author response:
**Experiments with smaller sampling size**

We thank you for your insightful suggestion and we conduct experiments with smaller sampling sizes (L). From the results in updated paper, Table 8 in Appendix A.3, the absolute value of performance drop is still not significant.
We believe there are two reasons for this phenomenon: (1) In the paper of vanilla self-consistency [2], the large improvements come from the comparison between single greedy sample and 40 samples. The author plotted the accuracy with respect to varying numbers of samples. Within a certain interval, the performance change is not significant either. (2) While the performance drop of Table 1 remains insignificant, the corresponding reduction of L decreases a lot compared with table 1, which proves that the interval with smaller L is more unsaturated. A quick preview is below:

|Method|CSQA L=10|CSQA L=20|GSM8K L=10|GSM8K L=20|
|:-:|:-:|:-:|:--:|:-:|
|SC|77.63|77.93|84.10|85.15|
|ESC|77.63|77.91|84.10|85.15|
|$\hat{L}$|6.47|8.60|7.09|10.14|
|$\hat{L}$-SC|77.39|77.55|83.31|84.10|

Reviewer comment (Presentation):
There are some issues in Table 1. For example, the SQA results of L-SC are generally much higher than SC, which look problematic. Also, it is confusing to list L in the table without additional notes, as L represents the sample size, while all other rows represent the task accuracies.

Author response:
**Fixing issues in Table 1**

We apologize that we have filled in the inconsistent values about SQA. That is because the candidate answer for question in SQA is either 'True' or 'False', but there will be some cases that the model generates neither of them. So there are two ways to vote the final answer: one is regarding these noisy answers as another type, e.g. '-1', to be included for voting; the other is removing this type of answers and only voting 'True' or 'False'. Our first choice was to vote based on the former criterion, but later we realized that the latter one made more sense, but we have forgotten to correct some values. We have fixed this problem in the revision.

As for the representation of L in Table 1, thanks for your advice and we have modified Table 1 to make it more reasonable for reading.

Reviewer comment (Experiments):
Show this comparison in Table 1 for smaller sampling sizes, e.g., those in Table 2, and see if the performance improvement achieved by ESC can be more significant.

Author response:
**Experiments with smaller sampling size**

We thank you for your insightful suggestion and we conduct experiments with smaller sampling sizes (L). From the results in updated paper, Table 8 in Appendix A.3, the absolute value of performance drop is still not significant.
We believe there are two reasons for this phenomenon: (1) In the paper of vanilla self-consistency [2], the large improvements come from the comparison between single greedy sample and 40 samples. The author plotted the accuracy with respect to varying numbers of samples. Within a certain interval, the performance change is not significant either. (2) While the performance drop of Table 1 remains insignificant, the corresponding reduction of L decreases a lot compared with table 1, which proves that the interval with smaller L is more unsaturated. A quick preview is below:

|Method|CSQA L=10|CSQA L=20|GSM8K L=10|GSM8K L=20|
|:-:|:-:|:-:|:--:|:-:|
|SC|77.63|77.93|84.10|85.15|
|ESC|77.63|77.91|84.10|85.15|
|$\hat{L}$|6.47|8.60|7.09|10.14|
|$\hat{L}$-SC|77.39|77.55|83.31|84.10|

Reviewer comment (Presentation):
Explain and fix issues in Table 1. For example, the SQA results of L-SC are generally much higher than SC, which look problematic. Also, it is confusing to list L in the table without additional notes, as L represents the sample size, while all other rows represent the task accuracies.

Author response:
**Fixing issues in Table 1**

We apologize that we have filled in the inconsistent values about SQA. That is because the candidate answer for question in SQA is either 'True' or 'False', but there will be some cases that the model generates neither of them. So there are two ways to vote the final answer: one is regarding these noisy answers as another type, e.g. '-1', to be included for voting; the other is removing this type of answers and only voting 'True' or 'False'. Our first choice was to vote based on the former criterion, but later we realized that the latter one made more sense, but we have forgotten to correct some values. We have fixed this problem in the revision.

As for the representation of L in Table 1, thanks for your advice and we have modified Table 1 to make it more reasonable for reading.