Paper: Are Human-generated Demonstrations Necessary for In-context Learning? (ICLR-2024)

Reviewer comment (Experiments):
While finding that it is feasible for a model to generate its own examples (rather than having human-crafted demonstrations) is insightful, the applicability of this method seems limited to large model sizes that are less likely to make reasoning errors during the self-generation process. How much would the method performance drop if tested with less capable language models? The paper would be strengthened with more thorough evaluations of when exactly SEC is usable.

Author response:
**Q1:** SEC’s performance on less capable language models.

  **A1:** We appreciate your concern about SEC’s applicability to smaller, less capable models. As reported in Section 4, SEC indeed shows a performance drop when applied to smaller models, which can struggle with instruction following and generating high-quality few-shot examples. This is a critical aspect of our ongoing research, and we plan to extend our evaluations to a wider range of models, including open-source ones. These additional experiments will provide deeper insights into SEC's scalability and effectiveness across different model sizes.

However, it is important to highlight the growing trend towards larger and more capable language models in the field. As language models continue to scale up in size and sophistication, we anticipate that the importance and relevance of methods like SEC will increase correspondingly. The ability of SEC to autonomously generate demonstrations is likely to become more valuable as models become more adept at complex reasoning and generation tasks. Therefore, while we acknowledge the current limitations with smaller models, we believe that the scaling trend in language models will make SEC an increasingly crucial method in the future.

Reviewer comment (Experiments):
As a baseline, it would be helpful to have direct comparisons with "Automatic chain of thought prompting in large language models. (Zhang et al., 2022)." to place this method in context relative to prior work. While the authors note that Auto-CoT makes use of questions from the training dataset as few-shot examples, it would still be insightful to see what performance gap this additional information leads to.

Author response:
We have carried out the experiments on Auto-CoT. CoT-SEC's performance is comparable to Auto-CoT, even without access to the full test dataset and additional clustering. 

|                        |   GSM8K  | ARC |
|---------------------- |------|-------|
|                       |      |       |
| *Zero-shot CoT*       |   73.4 | 84.1  |
| *CoT-ICL*             | 77.4 | 87.9  |
| *Auto-CoT*            |77.5|87.8|
| *CoT-SEC*             | 77.0|86.9|
|                      | (-0.5)|(-0.9) |

The results in the table will later be added into the revised version of our paper.

Reviewer comment (Writing):
Nit: Clean up Figure 1 further – e.g. typos follwing -> following , Demonetration -> Demonstration, in the Input to LLM section for SEC Demonstration Generation.

Author response:
**Q4:** Presentation and Figures  

**A4:** We thank the reviewer for pointing out the typographical errors in Figure 1 and the readability issue in Table 2. We will correct these in our revised version to enhance clarity and presentation quality.

Reviewer comment (Presentation):
Nit: Table 2 is a little hard to read, maybe have larger spaces / break lines between different rows of strategies.

Author response:
**Q4:** Presentation and Figures  

**A4:** We thank the reviewer for pointing out the typographical errors in Figure 1 and the readability issue in Table 2. We will correct these in our revised version to enhance clarity and presentation quality.

Reviewer comment (Experiments):
The authors note that ICL and SEC get different questions correct for GSM8K. Have they tried combining both methods? In other words, using ICL with human-crafted demonstrations combined with having the model additionally generate its own questions. Does this lead to better coverage in the correctly answered questions?

Author response:
**Q3:** Combining ICL and SEC

**A3:** The suggestion to combine ICL with human-crafted demonstrations and SEC’s model-generated questions is indeed intriguing. While we have not explored this combination, we recognize its potential for synergistically enhancing performance. We will initiate experiments combining these methods in the coming days. 

However, it's important to note that the primary contribution of this paper remains the introduction of SEC as a strong, novel and flexible zero-shot prompting method.

Reviewer comment (Experiments):
Using only a single LLM in the experiment makes it difficult to determine whether the quality of the results is based on the properties of the language model itself or whether it is simply a property specific to GPT 3.5 in its current version.

Author response:
We concur with the reviewer on the importance of investigating SEC's applicability to multiple models. To this end, we intend to extend our experiments to include both open-source models and the newer GPT-4 architecture.

Reviewer comment (Experiments):
Recent papers on prompting often analyze universal results by comparing multiple language models; as a science, it is important to show universal results in LLMs to some extent. The Chain-of-Thought paper also provides results for multiple LLMs.

Author response:
We concur with the reviewer on the importance of investigating SEC's applicability to multiple models. To this end, we intend to extend our experiments to include both open-source models and the newer GPT-4 architecture.

Reviewer comment (Evaluation):
Can you analyze a sufficient number of results to prove your hypothesis regarding the following question stated in your paper: "Why incorrect few-shot demonstrations could lead to correct final predictions, while correct few-shot demonstrations could also lead to incorrect predictions?"

Author response:
Sorry for the confusion. We acknowledge the reviewer's concern about the correctness of demonstrations. However, previous study shows that in-context learning does not rely on the input-label mapping in the demonstrations to perform the task (Min, Sewon, et al.). The correctness of individual demonstrations is not the sole determinant of the overall quality, especially in the context of language models' ability to generalize from these inputs.

Thus, regarding the phenomenon that correct answer rate increases even though the generated demonstration is incorrect, a reasonable explanation is that strict correctness may not be the primary influencing factor in ICL's performance. Instead, language models might utilize other cues and patterns within demonstrations to arrive at correct conclusions.

**Thus, the correctness of SEC's demonstration isn't a key factor of SEC's success.**

Moreover, while our study involved 20 correct and 20 incorrect predictions, each case incorporated 5 demonstrations, resulting in an effective analysis of 200 demonstrations. This sample size, we believe, offers a reasonable overview of potential error types within demonstrations. Nevertheless, we recognize the importance of expanding this analysis and plan to include a larger dataset in future work to enhance the robustness of our conclusions.

Reviewer comment (Experiments):
Can the generality of this result be analyzed using multiple language models?

Author response:
We have already carried out experiments of SEC and baselines on GPT4 and Llama2 34B. Please refer to the post with the title "Comparison between SEC and baselines on GPT4" and "Comparison between SEC and baselines on Llama2 34B".

Reviewer comment (Experiments):
The SEC method is only compared to the ICL approach where demonstrations are hand selected, i.e., not automatically selected. Several automated demonstration selection and curation approaches have been proposed in the last few years that should have been considered. SEC performs similar to hand-crafted ICL demonstrations. However, it is very likely that it might underperform once automated demonstration selection+curation approaches are introduced for comparison. Please look at the following papers and include them in your analysis:  
 a) https://aclanthology.org/2023.findings-acl.273.pdf  
 b) https://arxiv.org/abs/2211.04486  
 c) https://arxiv.org/abs/2310.10707  
 d) https://arxiv.org/abs/2302.05698  
 e) https://arxiv.org/abs/2104.08786

Author response:
Sorry for the confusion. First we want to clarify that SEC isn’t a variant of in-context learning. Inherently, SEC is a zero-shot method which only relies on test input.

**Q1**: Comparison with Automated Demonstration Selection Approaches

**A1**: We apologize for any confusion. We acknowledge the reviewer's point on comparing SEC with automated demonstration selection and curation approaches. However, it's crucial to emphasize that **SEC inherently functions as a zero-shot method**, relying solely on test input, which sets it apart from ICL. Comparing SEC directly with automated demonstration selection approaches may not be entirely fair, **as the latter often requires a substantial pool of training examples for selection**. Our intention in comparing SEC with ICL was primarily illustrative, highlighting SEC's superiority over the zero-shot baseline.

SEC substantially outperforms zero-shot baseline and reaches comparable performance with ICL methods which proof it’s efficacy. To provide a more comprehensive comparison, we are currently adding experiments with Zero-shot CoT. Preliminary results indicate that on benchmarks like GSM8k and ARC, Zero-shot CoT's performance (73.39 on GSM8k and 84.04 on ARC) is outperformed by CoT-SEC (77.0 on GSM8k and 86.9 on ARC).

For the five paper you kindly provided, (a) focuses on exemplar selection from the view of explanations, (b) focuses on active example selection, (c) focuses on applying ICL to offensive content paraphrasing, (d) focuses on CEIL, an ICL example selection method, and (e) focuses addressing the order sensitivity in ICL. These approaches, indeed, improve upon vanilla ICL using training datasets. However, **our approach with SEC offers a novel perspective by achieving comparable ICL performance without relying on training data**. This difference in approach underscores SEC's flexibility and ease of use, presenting an alternative direction in the realm of language model prompting.

Reviewer comment (Experiments):
Only closed-source GPT-based models from OpenAI were considered. The authors should investigate if SEC can be extended to open-source models.

Author response:
**Q3**: Applicability to Open-Source Models

Our initial study focused on closed-source GPT-based models due to their prevalent use and performance. However, we agree with the reviewer that investigating SEC's applicability to open-source models is essential. We plan to conduct additional experiments with open-source models to validate SEC's effectiveness across a wider range of language model architectures.

We have already carried out experiments of SEC and baselines on GPT4 and Llama2 34B. Please refer to the post with the title "Comparison between SEC and baselines on GPT4" and "Comparison between SEC and baselines on Llama2 34B".

Also, in all our main experiments, we included Zero-shot CoT as one of our baselines to better evaluate the superior of SEC over zero-shot methods. SEC consistently outperforms Zero-shot CoT on all tasks we examined. Please refer to the revised version of our paper and the results of GPT4 and Llama2 34B.

Reviewer comment (Novelty):
The main concern is the significance of the paper. The proposed prompting method can be treated as a two-step chain-of-though prompting by letting the LLM (1) first think about the possible demonstration and then (2) use the demonstration to do in-context learning. From this point of view, the prompting framework is one specific usage of CoT, which makes the contribution limited.

Author response:
**A1:** Your observation that SEC may seem akin to an extension of the chain-of-thought (CoT) prompting is insightful. However, SEC distinguishes itself by enabling the model to perform analogical reasoning, which is a leap beyond the capabilities of traditional CoT methods. SEC is fundamentally a zero-shot prompting pipeline. To provide a more comprehensive comparison, we are currently adding experiments with Zero-shot CoT. Preliminary results indicate that on benchmarks like GSM8k and ARC, Zero-shot CoT's performance (73.39 on GSM8k and 84.04 on ARC) is outperformed by CoT-SEC (77.0 on GSM8k and 86.9 on ARC).

These findings reinforce our belief that SEC's performance cannot be easily replicated by existing CoT methods in a Zero-shot scenario. SEC's novelty, therefore, lies in its unique ability to autonomously generate meaningful demonstrations, which sets it apart from existing CoT approaches.

Reviewer comment (Evaluation):
I am curious about the limitations of the proposed method. Using LLM for self-improvement may suffer from performance degradation – Once the LLM generates some wrong correction, the overall performance may drop significantly. Have you noticed any domains experiencing performance declines when using LLMs to generate their prompts?

Author response:
**A2:** We apologize for any confusion regarding the limitations of SEC. We recognize the potential for performance degradation with incorrect self-generated demonstrations. Our analysis indicates that large language models, particularly those with capacities at or above GPT-3.5, tend to be robust against minor errors in self-generated prompts for common tasks, maintaining performance comparable to that with human-crafted prompts.

Our experiments across various reasoning tasks show that SEC generally achieves promising results. However, we recognize that in rare or abstract reasoning tasks, which have limited representation in training data, the model might encounter challenges. We plan to conduct further experiments in these domains to comprehensively understand SEC's limitations. Notably, in the domain of MATH, where LLMs typically struggle, CoT-SEC has outperformed CoT-ICL, reaffirming our observation about the robustness of larger models against minor prompt errors.

In Section 4, we have evaluated SEC’s performance on smaller models and observed that its effectiveness diminishes with smaller model capacities. We are committed to extending our research with additional experiments using open-source models to further explore SEC’s applicability and limitations across different model scales.