From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper describes a way for improving the performance of a large language model through what amounts to a form of self-training. The model is used in question answering to generate high-confidence rational-augmented answers, which are then used to further fine-tune the model. The method improves performance on challenging datasets such as GSM8K, DROP, OpenBookQA, and ANLI without requiring extra supervision.

The paper shows strong empirical evidence that the method is effective despite its relative simplicity. This can inspire similar ways to teach large language models to self-reflect and improve.

The paper experiments only with a single proprietary LM, which means that results are not replicable. It is not clear whether the finding will be generally applicable to other LMs. The method is fairly straightforward and similar to methods that have been proposed in the past.
justification_for_why_not_higher_score: Lack of reproducibility -- all experiments are performed on a proprietary LM with no chance for anyone outside the particular organization to replicate them.

Results are shown at a single (very large) model scale. It is unclear whether they will hold for different scales as well.

The method is a rather straightforward application of similar methods that have been proposed in the past.

While there are ablations in the paper, their interpretations and significance are unclear.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper3654/Authors
title: Thanks for your reply!
comment: Dear Reviewer nphB,

Thank you again for your valuable feedback and comments! As the discussion period is ending soon, we would greatly appreciate it if you could let us know whether you are satisfied with our response. We will be happy to address any remaining concerns.

Sincerely, Paper3654 Authors

From: ICLR.cc/2023/Conference/Paper3654/Authors
title: Looking Forward to Your Reply!
comment: Dear Reviewer ombs,

Thank you again for your valuable feedback and comments! As the discussion period is ending soon, we would greatly appreciate it if you could let us know whether you are satisfied with our response. We will be happy to address any remaining concerns.

Sincerely,
Paper3654 Authors

From: ICLR.cc/2023/Conference/Paper3654/Authors
title: Response to Reviewer nphB
comment: Thanks for your constructive comments again!

> I agree that using more paths to get better performance is computationally very expensive and the proposed approach makes sense. However, in terms of novelty, this alone is very limited.

We agree that the approach is very straightforward. We also experimented approaches of using more sophisticated reasoning generation methods for training the model (e.g., using prompts to decompose questions into simpler sub-questions)  and saw promising results as well. We still decided to just use Self-Consistency to generate reasoning paths for training, because we want to keep the method simple and convey the idea of self-improving the reasoning ability of language models (both in-domain and out-of-domain) by training on reasoning paths (instead of answers).

Besides training on self-generated reasoning paths, we also propose other approaches which are novel and less explored by previous researchers.  
- Self-generating training questions as data augmentation (method in Sec. 3.3 Question Generation; experiment results in Sec 5.2 Self-Generating Questions)  
- Self-generating few-shot CoT prompts for zero-shot setting (method in Sec. 3.3 Prompt Generation; experiment results in Sec 5.2 Self-Generating Few-Shot CoT Prompts)  

> The plot on self-consistency with more paths make sense. Your training recipe has prompt variations as well. How would the plot change with varying the prompt formats, but still keep 160 paths?

The prompt variations in our training recipe are: (1) few-shot Chain-of-Thought (CoT) prompting; (2) few-shot standard prompting; (3) zero-shot prompting ending with "A: Let's think step-by-step."; (4) zero-shot prompting ending with "A:".

Among all the variations, (1) is the best prompting method to be integrated with Self-Consistency, because 
- (1) is better than (2) because it guides the model to generate a CoT reasoning path before generating the final answer, and this fits with the idea of Self-Consistency that a reasoning problem may have different solutions (different CoT paths to obtain the final answer).
- (1) is better than (3) and (4) because it provides real questions with correct reasoning examples to guide the language model generation. Results in [1] (Table 2) also show that Few-shot-CoT + Self-Consistency is better than Zero-shot-CoT + Self-Consistency.

We added those inferior prompts for training because we want the model to be trained on difficult tasks -- During training, the model tries to use simpler prompts like (2) (3) (4) to achieve results that only (1) can originally achieve. We didn't use the inferior prompts directly for inference because it will include more noises in the generated results.

[1] Large Language Models are Zero-Shot Reasoners. Neurips 2022. Kojima et al.  


> Since you are not using ground-truths, I am not sure what you meant by using external information. Those are just prompt variations. Without ground-truth or using external models/verifiers to get pseudo ground-truth label, training with any variation of input doesn't make the model "train/learn" anything better.

Sorry for the confusion. Our method doesn't use external models/verifiers to get pseudo ground-truth labels, but the self-generated CoT paths can be seen as pseudo labels (though in a sequence format instead of the final answer token). Our method is related to neural model self-training [1], and the main idea is to generate a set of pseudo labels in the target domain, and then re-train the network based on these pseudo labels. Through the self-training process, the model can gradually learn from high confident labels to correct the low confident labels. This technique is effective for various applications, such as in the field of computer vision [2].  

[1] Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks. Dong-Hyun Lee. 2013.   
[2] Confidence Regularized Self-Training. Zou et al. ICCV 2019.

From: ICLR.cc/2023/Conference/Paper3654/Reviewer_nphB
title: Thanks!
comment: Thanks for providing additional evidence. 
1. I agree that using more paths to get better performance is computationally very expensive and the proposed approach makes sense. However, in terms of novelty, this alone is very limited. 
2. The plot on self-consistency with more paths make sense. Your training recipe has prompt variations as well. How would the plot change with varying the prompt formats, but still keep 160 paths? 

> Our method does leverage external information, which is the training-set reasoning questions. They have similar distributions with test-time questions. The pre-trained language model may not be well-trained on reasoning questions, but through LMSI, the model “sees” a lot of training-set questions and becomes gradually good at making correct reasoning.

Since you are not using ground-truths, I am not sure what you meant by using external information. Those are just prompt variations. Without ground-truth or using external models/verifiers to get pseudo ground-truth label, training with any variation of input doesn't make the model "train/learn" anything better. It would just act as a filtering mechanism. Also check the experiment I mentioned in my Point-2 above.

From: ICLR.cc/2023/Conference/Paper3654/Authors
title: Response to Reviewer nphB
comment: Thanks for your valuable feedback! We update the paper and address your concerns as below:

>1. The paper presented an approach followed with empirical evidence, but did not provide why such an approach works in the first place.
> - The approach fine-tunes on the rationale-augmented reasoning paths using self-consistency, which means that self-consistency outputs without LMSI (fine-tuning approach) will be on par with CoT-Prompting with LMSI.  
> - The model is not learning any new information, the proposed approach is acting like a filtering method where the results achieved by a self-consistency can be achieved by fine-tuned LMSI in CoT setting. I think the self-consistency results for LMSI can be achieved by using more samples for self-consistency without LMSI

* Response to the first bullet:  
    - Self-consistency and LMSI can be applied together. We showed that LMSI + self-consistency achieved SOTA results that self-consistency alone cannot achieve (no matter how many paths were sampled). See the end of our response for empirical evidence.
    - Self-consistency saturates/converges as the number of sampled paths increases. At that point, only adding LMSI can further improve the results.
    - Self-consistency requires extensive computation – sampling 80 paths means 80 times the computation at test time, whereas our method does not increase cost at test time.

- Response to the second bullet:  
    - Our method does leverage external information, which is the training-set reasoning questions. They have similar distributions with test-time questions. The pre-trained language model may not be well-trained on reasoning questions, but through LMSI, the model “sees” a lot of training-set questions and becomes gradually good at making correct reasoning.

We hope our explanation is clear, and would like to know if you have any other comments on this.


> 2. Like I said, there were a lot of ablations in the paper where some of them proved some points presented in previous work (like Table-7) and others are new (Figure-3). On the new ones, the paper failed to provide good discussion on the outcomes/observations. For example, in Figure-3, both step-by-step and few-shot with step-by-step are converging for total samples at 80. Why is that the case?

The convergence of self-consistency is already discussed in the self-consistency paper [1]. If infinite reasoning paths were sampled, we would obtain the predicted probability distribution of the final answer marginalized over all the possible reasoning paths. In practice, Wang et al. [1] show that the performance of self-consistency saturates quickly with the increasing number of sampled paths, which agrees with our finding.

[1] Self-consistency improves chain of thought reasoning in language models. Wang et al. arXiv 2022.


>3. Please provide more discussion on whether self-improving a characteristic of large language models or it applies to any model? If yes/no, why? Any meaningful discussion would be helpful.

Our work relies on demonstration-based in-context few-shot learning abilities which only large models are able to perform [1]. A 6B model, GPT-J, only achieves 3.1% on GSM8K [2] with CoT prompting, while GPT-3 (175 B) achieves 46.9%, according to [3]. Thus, small models are not considered in our work.  

We conduct experiments on a recently released public model, UL2 (20 B) [4], and show results in Table 8 of Appendix A.1. Compared to the 540B model (decoder-only), UL2 has a different architecture (encoder-decoder) and pre-training objective. We observe that for most datasets, LMSI still improves the reasoning accuracy (+1.6% on DROP, +1.2% on OpenBookQA, and +0.7% on ANLI-A2), but the improvement on UL2 is not as large as that on 540B.

[1] Language Models are Few-Shot Learners. Brown et al. Neurips 2020.  
[2] Chain of thought prompting elicits reasoning in large language models. Wei et al. Neurips 2022.  
[3] STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning. Zelikman et al. Neurips 2022.  
[4] UL2: Unifying Language Learning Paradigms. Tay et al. Arxiv 2022.  


> 4. Can you check if baseline self-consistency with more examples can match LMSI or not?

We conduct experiments of self-consistency on the 540 B model without LMSI for up to 160 paths on GSM8K:  

| Path number |  Accuracy |
|:--------------|:-------------:|
| 40 | 74.6% |
| 80 | 75.2% |
| 120 | 75.5% |
| 160 | 75.6% | 

This result is still significantly worse than LMSI + self-consistency (82.1% with 32 paths on GSM8K).

From: ICLR.cc/2023/Conference/Paper3654/Authors
title: Response to Reviewer ombs
comment: We thank you for the constructive comments! We update the paper and address your concerns as below.

> 1. This paper uses all the methods of the past and proposes an obvious extension that takes the confident samples and trains over them. This is not novel at all with many researchers have thought about this but could not execute the idea because of a lack of resources, higher API costs, and non-availability of 540B LLMs.

We understand that LLM related research requires extensive resources that many researchers do not have access to. We conduct additional experiments on a publicly available 20B model: UL2 [4] (Table 8 in Appendix A.1).

Regarding the novelty claim:

First, we also propose the following novel approaches:  
- Self-generating questions then perform unsupervised training on self-generated questions (method in Sec. 3.3 Question Generation; experiment results in Sec 5.2 Self-Generating Questions)  
- Self-generating few-shot CoT prompts (method in Sec. 3.3 Prompt Generation; experiment results in Sec 5.2 Self-Generating Few-Shot CoT Prompts)  
- Improving small models from reasoning distillation (Sec. 5.3).

Second, our approach and results are not trivial, since:  
- Fine-tuning on reasoning paths (self-generated Chain-of-Thoughts) instead of on only the answers is crucial to achieving the claimed results (Paragraph: Importance of training with Chain-of-Thought formats).  
- The model’s reasoning ability can improve on Out-of-Domain tasks whose question styles are not seen during training. (Paragraph: Multi-task self-training for unseen tasks)  

We are fortunate to have enough resources for this work. As we work all together to advance NLP research, we believe that our findings are useful for all researchers, including the ones without extensive resources.

> 2. “Humans can look at a problem, and think of a solution many times, and the one that humans conclude the most is the most likely output and use that to think and train themselves to memorize that solution.” I don't agree with this point at all.

Our analogy is not scientific and we removed this sentence in the submission.


> 3. Research where such an important claim is being made (LLMs are able to self-improve) needs to be shown on multiple models (small to very large) and just showing the results on a 540B model does not prove the point that large LMs can self-improve by training on the samples that they produced.

Our work relies on demonstration-based in-context few-shot learning abilities which only large models are able to perform effectively [1]. A 6B model, GPT-J, only achieves 3.1% on GSM8K [2] with CoT prompting, while GPT-3 (175 B) achieves 46.9%, according to [3]. Thus, we focus on large language models.

We conduct experiments on a 20B model, UL2 [4], and show results in Table 8 of Appendix A.1. Compared to the 540B model, UL2 has a different architecture (decoder-only vs. encoder-decoder) and pre-training objective. We observe that for most datasets, LMSI still improves the reasoning accuracy (+1.6% on DROP, +1.2% on OpenBookQA, and +0.7% on ANLI-A2), but the improvement on UL2 is not as large as that on 540B. Please refer to Appendix A.1 for more details.


> 4. Finally the limitation of the experiments on one model (and that too not open sourced) is not enough to justify the big claim of "large language models can self improve". The reproducibility of the results is a concern to many.

We conduct experiments on a publicly available model UL2 [4], as shown in Table 8 of Appendix A.1.

[1] Emergent Abilities of Large Language Models. Wei et al. Arxiv 2022.  
[2] STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning. Zelikman et al. Neurips 2022.  
[3] Chain of thought prompting elicits reasoning in large language models. Wei et al. Neurips 2022.  
[4] UL2: Unifying Language Learning Paradigms. Tay et al. Arxiv 2022.

From: ICLR.cc/2023/Conference/Paper3654/Authors
title: Response to Reviewer hq7M
comment: Thanks for your positive feedback! We update the paper and address your concerns as below.


> 1. Can you apply the entire method to a smaller model say < 10B? 

Our approach relies on demonstration-based in-context few-shot learning which only works effectively on large language models, according to Wei et al. 2022 [1]. Additionally, Zelikman et al. 2022 [2] showed that a 6B model, GPT-J, achieves only 3.1% accuracy on GSM8K with few-shot CoT prompting, while GPT-3 (175 B) achieves 46.9%, according to [3]. Based on these existing studies, we believe that our approach does not apply on small models. We experiment on UL2 [4], a 20B publicly available model, as shown in the response of the next comment.  


> 2. It'd be great to see if these approaches can be used with language models that are available to everyone and at scales that most of the community can use.

We conduct additional experiments on a recently released public model UL2 (20B) [4] and show the results in Table 8 of Appendix A.1. We observe that for most datasets, LMSI still improves the reasoning accuracy (+1.6% on DROP, +1.2% on OpenBookQA, and +0.7% on ANLI-A2), although the improvement on UL2 is not as large as that on 540B.    


> 3. Replicating this is nearly impossible.

For reproducibility, we also experiment on a recently released public model UL2 [4]. Please refer to Appendix A.1 for the details of this experiment.

[1] Emergent Abilities of Large Language Models. Wei et al. Arxiv 2022.  
[2] STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning. Zelikman et al. Neurips 2022.  
[3] Chain of thought prompting elicits reasoning in large language models. Wei et al. Neurips 2022.  
[4] UL2: Unifying Language Learning Paradigms. Tay et al. Arxiv 2022.

From: ICLR.cc/2023/Conference/Paper3654/Reviewer_hq7M
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors propose an avenue to self-improve large language models without explicit supervision. They first generate candidates (in this case high-confidence rationale-augmented answers in their words) for unlabeled questions and further fine-tune the model on those unlabeled and self-generated solutions. They are able to improve performance on a very large 540B model on datasets that the community evaluates on such as GSM8K, DROP, OpenBookQA, and ANLI.
strength_and_weaknesses: Strengths: 
1) The empirical gains indicate that the method is effective.
2) The approach is reasonably straight forward, although difficult to estimate the impact of a variety of hyperparameters and choices (some of which are varied and shown in later in the paper) at each stage due to the complexity of fine-tuning such a large model.
3) One-type of distillation, fine-tuning a smaller model from the generations of the 540B model, is effective.

Weaknesses:
1) The method is only broadly applied to the 540B model that almost no one else has access to/can train or fine-tune. The distillation approaches show that the method is effective when using the samples from the 540B model. What happens if you don't have access to such a large model? Can you apply the entire method to a smaller model say < 10B? 
2) It'd be great to see if these approaches can be used with language models that are available to everyone and at scales that most of the community can use.
3) Replicating this is nearly impossible.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper reads well and understanding most of the paper is relatively easy.

Quality/Novelty: The authors integrate chain-of-thought prompting and self-consistency as means for self-improving large pretrained language models. Technically this basically just takes tools from other work and integrates them, but the evaluation and the utilization is reasonably novel.

Originality: I think many people in the community over the years have approached a self-supervised, self-improvement goal for AI systems and NLP systems, but this specific approach is relatively original and to my knowledge has not been tried before.

Reproducibility: Access to the pretrained model is not available, and the sheer size of the models make reproducibility impossible for the vast majority of labs, universities, and companies alike.
summary_of_the_review: The paper is good. The method is relatively straight forward and empirically the method performs well. The paper is clear and has additional experiments such as distillation that make it more compelling. 

There are limitations in that the models here are of squillions of parameters and not publicly available, so its nearly impossible to replicate. The method has not been tested on models that are of reasonable size or ones that are publicly available, which is a significant limitation. Given the compute resources at the authors' disposal, running a few additional experiments on relatively much smaller and publicly available models would strengthen the paper greatly and would be of significant interest to the community.

In general this paper is of broad interest to the NLP subcommunity at ICLR, so I vote that it be accepted.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper3654/Reviewer_ombs
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper discusses that training using only unlabelled data can improve large language models' (LLMs) reasoning capabilities. In other words, the authors used an LLM (PaLM in this case), generated multiple outcomes for a given problem, and used the ones where the model was “highly confident” to retrain the model as target outputs to achieve better results than the non-trained version. The authors argue that unlabelled fine-tuning can lead to self-improvement of LLMs. 
strength_and_weaknesses: ### Strengths:
- The work pushed the state-of-the-art results on multiple datasets including GSM8K which seems to be a challenging dataset even for larger language models.
- Using what the model knows (in this case by predicting multiple answers to the same question, and doing majority voting to estimate the possible correct output) to train the model further can help in adapting LLM to a specific task and can be used in future to train LLMs without any supervision. 

### Weaknesses: 
- Prompting is a popular technique for querying LLMs because of their size and training them is not efficient/possible in most scenarios. Various prompting strategies have been developed in the past that have improved the performance of LLMs significantly, like Chain of thoughts, appending “let's do this step by step”, and so on. An obvious extension was to generate multiple times using the same/different strategies, and select the one with higher confidence or cluster them or do some majority voting. This paper uses all the methods of the past and proposes an obvious extension that takes the confident samples and trains over them. This is not novel at all with many researchers have thought about this but could not execute the idea because of a lack of resources, higher API costs,  and non-availability of open-source LLMs like PaLM.
- Secondly, proposing an approach and claiming that “This is similar to how a human brain sometimes learns: given a question, think multiple times to derive different possible results, conclude on how the question should be solved, and then learn from or memorize its own solution” is just random with not theory or study to back this up. In other words, it is saying that humans can look at a problem, and think of a solution many times, and the one that humans conclude the most is the most likely output and use that to think and train themselves to memorize that solution. I don't agree with this point at all (of course I am happy to listen to why authors thought about this). 
- Research where such an important claim is being made (LLMs are able to self-improve) needs to be shown on multiple models (small to very large) and just showing the results on a 540B model does not prove the point that large LMs can self-improve by training on the samples that they produced. When generating outputs with let's say a smaller model like T5-large, where the accuracy will not be that high, the model outputs cannot be used to train them as most of the time, the answers will be incorrect. So my guess is that this approach only works for PaLM or similar-sized models. 
- Finally the limitation of the experiments on one model (and that too not open sourced) is not enough to justify the big claim of "large language models can self improve". The reproducibility of the results is a concern to many. 


clarity,_quality,_novelty_and_reproducibility: - The paper is clearly written and very easy to follow. The quality of the paper is very sound and experiments are conducted to support the claim (but restricted to one LLM only). 
- The paper is original in its implementation but not in ideation (very obvious extension that was not conducted by a lot of researchers due to lack of resources). 
- However, the results cannot be reproduced due to the lack of open source policy for PaLM by google and there is no way to test if the results are accurate or not. 
summary_of_the_review: The authors proposed a way to self improve large language models. This claim is very not supported by the experiments where the results are shown for just one LM and that too is not openly available. So the validity of the method cannot be tested/justified by the community. Moreover, the approach is not novel in its ideation but only novel in its implementation (as the method was not implemented by others due to resource constraints) and hence I would suggest showing the novelty of the idea on more models and datasets to justify the claims. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3654/Reviewer_nphB
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper demonstrates that large language models (LLMs) can be self-improved by generating high-confidence rationale augmented answers for unlabelled questions using an LLM and use that data to fine-tune the same LLM to improve itself. The paper empirically shows that this approach improves the reasoning abilities of the LLMs on various datasets in both in-domain and out-of-domain settings. Further, a broad range of ablations are conducted to show that fine-tuning on reasoning is critical for self-improvement.
strength_and_weaknesses: 
Strengths: 
1. The paper is well written and easy to follow. 
2. Evaluated the approach of a lot of datasets and showed improvements.
3. Lot of ablations are conducted which are very useful. 

Weaknesses: 
1. The paper presented an approach followed with empirical evidence, but did not provide why such an approach works in the first place (i.e., approach is not backed well enough). Following are my concerns on the approach:
   - The approach fine-tunes on the rationale-augmented reasoning paths using self-consistency, which means that self-consistency outputs without LMSI (fine-tuning approach) will be on par with CoT-Prompting with LMSI. This is more or less consistent with the observations in main Table-3. 
   - As per my understanding, the model is not learning any new information, the proposed approach is acting like a filtering method where the results achieved by a self-consistency can be achieved by fine-tuned LMSI in CoT setting. I think the self-consistency results for LMSI can be achieved by using more samples for self-consistency without LMSI. These experiments are not there in the paper. 
2. Like I said, there were a lot of ablations in the paper where some of them proved some points presented in previous work (like Table-7) and others are new (Figure-3). On the new ones, the paper failed to provide good discussion on the outcomes/observations. For example, in Figure-3, both step-by-step and few-shot with step-by-step are converging for total samples at 80. Why is that the case? 


clarity,_quality,_novelty_and_reproducibility: 1. Please provide more discussion on whether self-improving a characteristic of large language models or it applies to any model? If yes/no, why? Any meaningful discussion would be helpful. 
2. Can you check if baseline self-consistency with more examples can match LMSI or not?
summary_of_the_review: Overall, the paper has many interesting experiments but lacks proper evidence and discussion on why this approach works. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough