Paper: Chain-of-Thought Reasoning is a Policy Improvement Operator (ICLR-2024)

Reviewer comment (Experiments):
On page 7, paragraph 2, the authors claim "we use K=5, which was found to be a good balance between computational speed and accuracy". This is an important claim that greatly impacts their results, but no data is provided to support this claim.

Author response:
**Regarding ablations on simplify-and-guess.** In Figure 5, we run the experiment measuring the impact of using K=0 (no decomposition and just chess) as well as K=N-1 (break the problem all the way down) compared to a middle ground of K=5. We will run an ablation measuring the impact of other intermediate values of K shortly.

Reviewer comment (Presentation):
Figure 6 is a key figure for the paper, yet the accuracy cannot be meaningfully read from these bars.

Author response:
**Regarding Figure 6.** Thank you for pointing out the difficulty in reading accuracy in this figure. In the updated paper, we include a table of all accuracies in the Appendix for ease of reading. Our hypothesis for why there might be better generalization in the middle than at the ends is that the model may be reaching its capacity for learning (see general comment) as it allocates more and more of its “ability” to fast-adding larger and larger digit numbers.

Reviewer comment (Evaluation):
Figure 6 shows better behavior in the middle than on the ends.

Author response:
**Regarding Figure 6.** Thank you for pointing out the difficulty in reading accuracy in this figure. In the updated paper, we include a table of all accuracies in the Appendix for ease of reading. Our hypothesis for why there might be better generalization in the middle than at the ends is that the model may be reaching its capacity for learning (see general comment) as it allocates more and more of its “ability” to fast-adding larger and larger digit numbers.

Reviewer comment (Novelty):
The authors should identify the key data efficient and curriculum learning references that could be used to improve these results.

Author response:
**Improving Data Efficiency.** In Appendix I, we find that doubling the parameter count of the model leads to almost a 10x reduction in data needed. An interesting  direction for future work is to see if this trend continues.

Reviewer comment (Theory):
The authors propose hypotheses that, in my opinion, are inappropriate. For instance, the authors mention that "a sufficiently large pre-trained language model might be able to forgo the supervised training period entirely and begin self-training immediately, perhaps with only a few examples of in-context demonstrations." It would be safer to avoid phrasing it like this as there is no supporting evidence from relevant references or experimental results.

Author response:
**Hypothesis on Emergence.** The reviewer took issue with our hypothesis that a sufficiently large LLM might be able to forgo supervised training entirely. This was framed as a hypothesis, as we did not have the resources to run this particular experiment. However, we politely disagree that there is no supporting evidence in the paper. Specifically, the preceding sentence refers to experiments done in Appendix I in which we found that that doubling the parameter count of the pretrained model (from 300M to 582M) reduced the training data needed in the supervised training period by almost a factor of 10 (from 4M to 0.5M supervised training examples). We did not have the computational resources to do an extensive study of scaling laws, but believe it is a reasonable hypothesis that a sufficiently large model might reduce to only a few examples. In particular, the SOTA LLMs have 1000x (~2^10) the parameter count of the models which we used here. Additionally, the history of LLMs are filled with tasks that previously required fine-tuning but can now be done with mere prompting (including chain-of-thought reasoning itself!). However, given this feedback, we have phrased our hypothesis more cautiously in the updated version of the paper.

Reviewer comment (Reproducibility):
How is the supervised training dataset generated? What distribution of prompts is used for answer generation during self-training? Approximately how many tokens are used for training in each phase?

Author response:
**Supervised training Questions.** In Figure 2, we provide explicit examples of the data for both fast and slow addition. These examples can be easily generated programmatically and the script used to generate them is provided in the supplementary data in   generate_data.py.  N-digit addition problems are generated by randomly sampling two N-digit numbers. Appendix I provides a graph depicting how much data is used during the supervised/self-train period. In particular, Figure 10 shows that the 582M model used around 0.5M training examples during the supervised training period and self-generated an additional 1M training examples for supervised training. The number of tokens per example varies but is on the order of 100 for slow addition (per decomposition step) and 10 for fast addition.

Reviewer comment (Experiments):
How does curriculum learning contribute to final performance? I see no results in Appendix J.

Author response:
**Curriculum Learning.** Appendix J states that: “We run an ablation where we train a 582M parameter ByT5 model on 1 through 6 digit addition in a single supervised fine-tuning step, instead of via the curriculum learning setup done in Section 3.5. We find that this model, when properly trained, generalizes to up to 9 digit (slow) addition perfectly.” Our conclusion is that curriculum learning is not necessary in the  supervised training, but required in the self-training phase, as the model cannot generate the data for N+1 digit addition until it has learned to add N digit numbers. In the paper, we used curriculum learning in both phases because this provides that the unique difference between self-training and supervised training was the data generation.

Reviewer comment (Experiments):
This article lacks specific details regarding the dataset used, including the sizes of the training and testing sets, as well as the methodology for constructing the dataset (including both fast and slow datasets).

Author response:
**Regarding dataset details.** The reviewer claims that the paper lacks details about the dataset. We respectfully disagree. In Section 3.2, we state that “all training examples are generated programmatically by an external script during supervised learning.” This script is included in the supplementary data in the file called generate_data.py. In Figure 2, we provide explicit examples of the data for both fast and slow addition. Additional details about the dataset, including the precise size of the dataset and the split between validation and training, are given in Appendix B.

Reviewer comment (Writing):
Figure 4 and the simplify-then-guess process is rather confusing. Can the authors provide a more detailed explanation or illustrative example of the simplify-then-guess process? (see question1,2).

Author response:
**Fast adding / simplify-and-guess.** Fast adding indeed means generating an answer without using chain-of-thought. Figure 2 contains specific examples for both slow and fast addition. To describe simplify-and-guess, it is helpful to walk through Figure 4. Given an eight digit problem, the model first breaks down the problem into a seven digit problem (slow chain-of-thought), before taking a guess for that seven digit problem (fast addition). This is guess #1. To obtain further guesses, the model breaks down the 7 digit problem it produced in the previous step into a 6 digit problem, before taking a guess to obtain guess #2. This is repeated K times and all K guesses are aggregated via majority vote.

Compared to simply decomposing the problem until it is 1-digit, simplify-and-guess is an additional error- checking mechanism in case there is error introduced in the decomposition step. Figure 5 describes the benefit that simplify-and-guess provides over just simplification or just guessing. Additionally, we hope to include an ablation explorating the impact of different choices of K shortly.

Reviewer comment (Presentation):
The article needs further improvement in terms of its presentation and image formatting. Some images are extending beyond the text width, which affects the overall layout and readability.

Author response:
We thank the reviewer for their comments. Regarding the comments on formatting, these have been fixed in the updated version of the paper.

Reviewer comment (Experiments):
In section 3.3.1, when saying “taking the first guess of a solution”, is the solution to the 8-digit problem or the 7-digit problem? If all the guesses belong to the simplified problem, how the majority voting is conducted? Do the authors sample different answers for each sub-problem and conduct majority voting separately?

Author response:
**Fast adding / simplify-and-guess.** Fast adding indeed means generating an answer without using chain-of-thought. Figure 2 contains specific examples for both slow and fast addition. To describe simplify-and-guess, it is helpful to walk through Figure 4. Given an eight digit problem, the model first breaks down the problem into a seven digit problem (slow chain-of-thought), before taking a guess for that seven digit problem (fast addition). This is guess #1. To obtain further guesses, the model breaks down the 7 digit problem it produced in the previous step into a 6 digit problem, before taking a guess to obtain guess #2. This is repeated K times and all K guesses are aggregated via majority vote.

Compared to simply decomposing the problem until it is 1-digit, simplify-and-guess is an additional error- checking mechanism in case there is error introduced in the decomposition step. Figure 5 describes the benefit that simplify-and-guess provides over just simplification or just guessing. Additionally, we hope to include an ablation explorating the impact of different choices of K shortly.

Reviewer comment (Writing):
What is fast add? Does it mean generating the answer directly without using CoT (slow)?

Author response:
**Fast adding / simplify-and-guess.** Fast adding indeed means generating an answer without using chain-of-thought. Figure 2 contains specific examples for both slow and fast addition. To describe simplify-and-guess, it is helpful to walk through Figure 4. Given an eight digit problem, the model first breaks down the problem into a seven digit problem (slow chain-of-thought), before taking a guess for that seven digit problem (fast addition). This is guess #1. To obtain further guesses, the model breaks down the 7 digit problem it produced in the previous step into a 6 digit problem, before taking a guess to obtain guess #2. This is repeated K times and all K guesses are aggregated via majority vote.

Compared to simply decomposing the problem until it is 1-digit, simplify-and-guess is an additional error- checking mechanism in case there is error introduced in the decomposition step. Figure 5 describes the benefit that simplify-and-guess provides over just simplification or just guessing. Additionally, we hope to include an ablation explorating the impact of different choices of K shortly.

Reviewer comment (Reproducibility):
This paper mentioned that "Training examples for each type are prefixed with a special token depicting their type, so the model can differentiate between the two." Could the authors please clarify what special token is being used for this purpose?

Author response:
**Data Prefix.** Training examples were prefixed with either “Add fast.” or “Add slow.” However, since the model was fine-tuned (and not just prompted) on these examples, the precise prefix is unlikely to be important since the model can easily learn the mapping between the prefix and the task at hand.

Reviewer comment (Novelty):
There are some missing citations e.g. in the self-improvement of LLMs space (https://arxiv.org/abs/2309.03409, https://arxiv.org/abs/2211.01910, https://arxiv.org/abs/2309.16797) and in the fine-tuning of LLMs via prompted LLMs space (https://arxiv.org/abs/2212.08410, https://arxiv.org/abs/2212.10071).

Author response:
**Regarding related work.** We don’t see much connection between the papers on distilling chain-of-thought utterances from a larger teacher model into a smaller student model and our  work, as we focus on self-learning where a model learns to self-improve without access to any external data source (whether from a larger teacher model or ground truth data). However, the other suggested related work is very relevant and included in section in the updated version of the paper with references to self-improvement via prompting.

Reviewer comment (Evaluation):
Does the fine-tuned model lose the "general" capabilities that make it so valuable in the first place? The authors do not verify that fine-tuning does not reduce performance on standard LLM benchmarks. Can they comment on this?

Author response:
**Regarding loss of existing capabilities.** We do not claim that SECToR produces an LLM that is superior to the original one -- rather, our goal is  to establish a proof-of-concept that self-learning in language models -- for multiple iterations that extend far beyond the original distribution of training -- is empirically possible. As per our general response, above, the paper’s primary goal is not to teach an LLM to perform addition — one could simply use supervised fine-tuning for this. The goal is to provide a proof of concept for self-learning, with  addition used as a testbed.

Reviewer comment (Experiments):
In Section 3.3.1 it is very unclear where the dataset of problems with N digits comes from. Is this generated by the large language model itself, or is this pre-generated synthetically? If the latter, the utility of the method is reduced, because the method is no longer self-contained. Could the authors clarify?

Author response:
**Regarding problem sources.** In SECToR, the problem statements themselves were provided to the model and not self-generated. In future work, we will explore asking the model to generate the problems themselves as well as the candidate solutions.

Reviewer comment (Experiments):
In Section 3.3.1 it is unclear why the iterative method of "simplify then guess" is required. If the chain of thought prompt reduces N+1-digit addition to N digit addition, and the fast addition for N digits is highly reliable, then surely further decomposition is not required? It would be useful to see an ablation of the full "simplify then guess" method, compared with the simpler "one step" one which I have suggested here.

Author response:
**Regarding the importance of simplify-and-guess.** The reviewer’s proposed method would correspond to using simplify-and-guess with K=1 (simplify once then guess immediately). The summary is that although decomposition and fast-adding are fairly reliable, even a tiny error incurred in the data generation process is amplified in future iterations due to error avalanching. We will perform an ablation on the precise importance of K and include it in the next draft of our paper.