Paper: Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution (ICLR-2024)

Reviewer comment (Experiments):
While Promptbreeder is an important approach as the prompt strategies, it is complex in its composition such as a mutation prompt, a hyper mutation prompt, a domain-specific problem description, and a seed thinking-styles, and then lacks the ablation analysis to show which components are effective and how effective they are.

Author response:
**Ablation Analysis:**

You state that our paper “lacks the ablation analysis to show which components are effective and how effective they are”. You will be glad to hear that our original submission already contained a comprehensive ablation analysis in Appendix L, however, we have moved this to Appendix B as it was easily overlooked. It shows that every component of Promptbreeder is important, and the interplay of different mutation operators is what makes Promptbreeder effective. We agree that this ablation analysis is crucial to show which components of Promptbreeder are effective, so to decrease the chances of other readers overlooking this important part of our work, we have made the reference to Appendix B in bold in the main text. We thank you for bringing this to our attention and improving the paper in this way.

Reviewer comment (Novelty):
Promptbreeder appears to rely on past prompt strategies or their combination, lacks its motivation and theoretical considerations, and lacks sufficient experimental results to support them.

Author response:
**Theoretical Motivation:**

Your observation on Promptbreeder's reliance on past prompt strategies is valuable. We designed Promptbreeder to not only incorporate but also evolve beyond these strategies. The addition of genetic algorithms to our framework is grounded in existing literature, advocating for diverse and non-greedy exploration that circumvents local optima. We've elaborated on this in the revised draft, highlighting how Promptbreeder transcends mere aggregation of existing strategies. Control 1 in our paper demonstrates that initializing task prompts with these strategies is less effective than employing evolutionary evaluations.  

**Comprehensive Comparative Study:**
We respectfully disagree with your claim that our paper lacks sufficient experimental results. Promptbreeder was benchmarked against various algorithms across nine distinct problems, in fact going beyond the standards as seen in published state-of-the-art work like APE [1] and Plan-and-Solve [2]. However, to address your point, we've expanded the comparison in Table 1 of our revised draft even further, adding three more control groups to reinforce the validity of Promptbreeder.

[1] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2023). Large Language Models Are Human-Level Prompt Engineers. arXiv. https://doi.org/10.48550/arXiv.2211.01910

[2] Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023). Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. arXiv. https://doi.org/10.48550/arXiv.2305.04091

We also point to the fact that Promptbreeder outperforms APE on 21/24 instruction induction tasks which are outlined in the Appendix L.

Reviewer comment (Experiments):
As authors use only two LLMs as baselines, the generality of the proposed method cannot be determined.

Author response:
**Sensitivity to existing strategies and types of LLM**: We would like to point you to Appendix B (Ablation Analysis) and Appendix M (GPT-3.5 Results). The results in these two appendices demonstrate that Promptbreeder is sensitive to the removal of mutation operators (Appendix B), but it is not sensitive to changing the underlying LLM (Appendix M). That is, Promptbreeder is still effective when replacing PaLM 2-L with GPT-3.5. Thank you for raising this—we will discuss it in more depth in the main paper.

Reviewer comment (Experiments):
What is the rationale for the baseline selection in Table 1?

Author response:
**Rationale for Baseline Selection:**

The selection of baselines was driven by a desire to compare Promptbreeder with the latest state-of-the-art prompting strategies. Additionally, two control experiments were conducted to validate the efficacy of evolutionary processes in Promptbreeder over random search and to demonstrate improvements over the initial problem descriptions used as prompts.

Reviewer comment (Experiments):
Can you show how effective it is compared to LLaMA or OPT as baselines?

Author response:
**Sensitivity to existing strategies and types of LLM**: We would like to point you to Appendix B (Ablation Analysis) and Appendix M (GPT-3.5 Results). The results in these two appendices demonstrate that Promptbreeder is sensitive to the removal of mutation operators (Appendix B), but it is not sensitive to changing the underlying LLM (Appendix M). That is, Promptbreeder is still effective when replacing PaLM 2-L with GPT-3.5. Thank you for raising this—we will discuss it in more depth in the main paper.

Reviewer comment (Evaluation):
The proposed PB algorithm appears to rely heavily on interactions with the LLM compared to the baselines. As a result, solely evaluating its performance based on accuracy may not provide a fair assessment of its capabilities.

Author response:
We appreciate your point about Promptbreeder’s interaction intensity with LLMs. As a general-purpose self-referential self-improvement mechanism, Promptbreeder leverages LLMs to mutate a population of task-prompts, evaluating their fitness on a training set over multiple generations. This process inherently involves significant interaction with LLMs, but it is crucial for the evolutionary approach that underpins Promptbreeder's effectiveness​​. The sample efficiencies of the algorithm for each dataset are shown in the Appendix K for each problem. For example, on AQuA the best individual was produced after 1400 mutations which equates to 1400 x 2 x 100 Q:A pairs tested. While this may seem costly at first, we believe Promptbreeder will have important practical implications as the time and cost to find a superior prompt is a one-time cost for a given domain, and amortized with respect to many LLM inferences at deployment time of the prompt.

Reviewer comment (Theory):
While the authors acknowledge that hand-crafted prompt-strategies are often sub-optimal, they do not offer a guarantee or highlight any asymptotic properties for the PB algorithm, leaving room for uncertainty regarding its long-term effectiveness.

Author response:
Note that such guarantees are generally not available for evolutionary algorithms in realistic large and noisy problem spaces. However, we have demonstrated empirical superiority to other optimization techniques such as APE, random task prompt initialization (see our own controls in Table 1), and ORPO on GSM-8k. We have shown that using the diverse set of initial task prompts improves Promptbreeder’s search for effective prompts in various domains.

Reviewer comment (Presentation):
The main text of the paper is relatively concise, with several crucial aspects relegated to the appendix. This arrangement can disrupt the smoothness of the reading experience.

Author response:
Thank you for your suggestion to improve our paper. We have moved Figure 3 into the main paper in improved form.

Reviewer comment (Evaluation):
I am quite curious about the sample efficiency of the algorithm, as evolutionary algorithms often suffer from poor sample efficiency.

Author response:
We appreciate your point about Promptbreeder’s interaction intensity with LLMs. As a general-purpose self-referential self-improvement mechanism, Promptbreeder leverages LLMs to mutate a population of task-prompts, evaluating their fitness on a training set over multiple generations. This process inherently involves significant interaction with LLMs, but it is crucial for the evolutionary approach that underpins Promptbreeder's effectiveness​​. The sample efficiencies of the algorithm for each dataset are shown in the Appendix K for each problem. For example, on AQuA the best individual was produced after 1400 mutations which equates to 1400 x 2 x 100 Q:A pairs tested. While this may seem costly at first, we believe Promptbreeder will have important practical implications as the time and cost to find a superior prompt is a one-time cost for a given domain, and amortized with respect to many LLM inferences at deployment time of the prompt.

Reviewer comment (Theory):
It will be nice to provide a guarantee or asymptotic property for the PB.

Author response:
Note that such guarantees are generally not available for evolutionary algorithms in realistic large and noisy problem spaces. However, we have demonstrated empirical superiority to other optimization techniques such as APE, random task prompt initialization (see our own controls in Table 1), and ORPO on GSM-8k. We have shown that using the diverse set of initial task prompts improves Promptbreeder’s search for effective prompts in various domains.

Reviewer comment (Experiments):
The proposed method Promptbreeder still requires initial information for specific task (like description or mutation prompts), where worse initialization may lead to worse performance. This makes the method may not generalize to various tasks.

Author response:
**Concerning the Requirement of Initial Information:**

You rightly pointed out that the effectiveness of Promptbreeder depends on the initial information provided, such as task descriptions or mutation prompts. To address this, we have included a new section in the Appendix of our revised draft and carried out two more control experiments showing that even with poor and general problem descriptions, PB shows consistent improvements in fitness. 

- **Generation of Thinking Styles and Mutation Prompts from Problem Descriptions:** We have developed a method by which thinking styles and mutation prompts can be autonomously generated from the problem description itself, reducing the dependency on external initialization.
- **Concerns about the initial problem description information provided to Promptbreeder:** We demonstrate (Appendix N) that even with neutral or misleading problem specification (task descriptions) Promptbreeder achieves substantial improvement in terms of the test set fitness of evolved prompts.

We respectfully disagree with the implication that our method may not generalize across various tasks. The aforementioned enhancements and experimental results strongly suggest that Promptbreeder is not only adaptable but also capable of performing effectively across a diverse range of tasks, even with minimal or suboptimal initial information. Furthermore, the fact that Promptbreeder outperforms APE on 21/24 instruction induction tasks also highlights this, see Appendix L.

Reviewer comment (Experiments):
PROMPTBREEDER aimed to concurrently refine both task and mutation prompts, considerably expanding the search space. However, the absence of navigation during each evaluation often resulted in unpredictable performance for the successive generated prompts. This is evidenced by the persistence of less effective prompts after extensive evaluations, as illustrated in Figure 3.

Author response:
**Absence of Navigation:**

The absence of directed navigation is an important component of an evolutionary algorithm. The success of evolution relies on sequences of random mutations that, when accumulating over time, lead to increased fitness. Promptbreeder, an evolutionary algorithm, makes use of this property. In fact, even keeping around less fit (but less specialized) individuals in an evolving population can be important for the entire evolutionary system to arrive at better solutions in the long term [1]. A more directed Promptbreeder approach is conceivable, but it would move Promptbreeder away from an evolutionary approach which is outside of the scope of our paper.

[1] Sudholt, D. The Benefits of Population Diversity in Evolutionary Algorithms: A Survey of Rigorous Runtime Analyses, 2018. https://arxiv.org/abs/1801.10087

Reviewer comment (Reproducibility):
It is not clear how the "Mutator Prompts" (Table 2) and "Thanking Styles" (Section D) are created. Are they derived from pre-existing prompt strategies? Are these prompts hand-crafted?

Author response:
**Origin of "Mutation Prompts" and "Thinking Styles":**

We appreciate your inquiry about the creation of the "Mutation Prompts" and "Thinking Styles." These were hand-designed for general problem-solving domains and are thus widely applicable (as demonstrated by our empirical gains on multiple diverse reasoning benchmarks). We have added a new section (Appendix G) detailing how an LLM can generate these lists from a problem description using a hierarchical generation method.

Reviewer comment (Experiments):
This study exclusively shows the performance of PROMPTBREEDER with PaLM 2-L, raising questions about its generalization ability to other LLMs. Specifically, whether PROMPTBREEDER method can be effectively utilized to enhance prompts for LLMs? Whether the prompts refined using PROMPTBREEDER in conjunction with PaLM 2-L can yield improved results when employed with alternative LLMs.

Author response:
**Generalization:**

Many thanks for the expressing the concern that this study exclusively shows the performance of PROMPTBREEDER with PaLM 2-L. To address this we ran PB on GSM-8k using GPT3.5-Turbo achieving test set scores of 65.5% and  63.9% accuracy respectively. Results are shown in Appendix M. The PS+ prompt only achieved 44.7% with GPT-3.5-Turbo. This provides evidence that PB generalizes to other language models. 

**Transfer of Promptbreeder Prompts to other LLMs:**

To address your question about Promptbreeder's applicability to other LLMs, note that we do not claim prompt transferability across LLMs—Promptbreeder is designed to find prompts that are tailored to the unique characteristics of each LLM for optimal task performance through automated prompt engineering. We will clarify this in a revised version of the paper.

Reviewer comment (Presentation):
In Figure 3, the y-axis label is not visible. Additionally, what is the relationship between "number of evaluations" and "number of generations"? It is confusing since Section 4 reported that the populations "evolved for typically 20-30 generations," but there are 2000 evaluations in Figure 3.

Author response:
**Clarifications on Figure 3, OPRO Evaluation, and Color Coding:**

We have corrected and enhanced Figure 3, providing clarity on the relationship between evaluations and generations (a generation being POP_SIZE evaluations). Regarding OPRO's evaluation on GSM8K, we suggest the authors of OPRO may consider evaluating it on a broader range of tasks, as we have done in our study. We have standardized the color coding for task-prompts and mutation-prompts throughout the paper to enhance readability and coherence.

Reviewer comment (Experiments):
Please clarify why OPRO was only evaluated on GSM8K in Table 1.

Author response:
**Clarifications on Figure 3, OPRO Evaluation, and Color Coding:**

We have corrected and enhanced Figure 3, providing clarity on the relationship between evaluations and generations (a generation being POP_SIZE evaluations). Regarding OPRO's evaluation on GSM8K, we suggest the authors of OPRO may consider evaluating it on a broader range of tasks, as we have done in our study. We have standardized the color coding for task-prompts and mutation-prompts throughout the paper to enhance readability and coherence.

Reviewer comment (Presentation):
For the sake of readability, it would be beneficial if the color coding is consistent across different figures and texts. For example, the "mutation prompt" is color-coded as red in the text on Page 6, yet appears in shades of blue (and not the exact same blue) in Figures 1 and 2.

Author response:
**Clarifications on Figure 3, OPRO Evaluation, and Color Coding:**

We have corrected and enhanced Figure 3, providing clarity on the relationship between evaluations and generations (a generation being POP_SIZE evaluations). Regarding OPRO's evaluation on GSM8K, we suggest the authors of OPRO may consider evaluating it on a broader range of tasks, as we have done in our study. We have standardized the color coding for task-prompts and mutation-prompts throughout the paper to enhance readability and coherence.

Reviewer comment (Presentation):
Some result sections in appendix should be moved to the main text as it really helps to show how the algorithm improve the prompts and how the prompts look in the end.

Author response:
**1.& 2. Prompt Space Exploration in Evolution Algorithms:**

We acknowledge your concern regarding the adequacy of evolution algorithms for exploring the large prompt space in Large Language Models (LLMs). To address this, we have updated Figure 3 (and, as suggested, moved it from the Appendix to the main part of the paper—thank you for the suggestion!) to showcase not only the highest-performing ('elite') prompts but also a variety of low-fitness prompts generated during the exploration phase. This figure shows that there is in fact considerable exploration in prompt space, with many prompts being produced that are very different to the elite prompt. Elite prompts also change in subtle ways throughout the evolutionary run and achieve much higher fitness at the end. The final prompt is "Sentences are given, and a single word. The output should indicate whether the given word has the same sense in the two given sentences, yes or no." with an early elite prompt in the run being "I'll give you two sentences and a word. Your task is to write if the meaning of the word is the same in both sentences or not."

Reviewer comment (Novelty):
I am afraid that I am not familiar with evolution algorithm literature but I personally feel the overall novel comtribution of this paper is limited as it seems all mutation operators are pretty standard, even those "self-referential" ones. And the backbone evolution algorithim seems very simple and out of box.

Author response:
**3. Simplicity and Novelty of Mutation Operators:**

Although some of the mutation operators in Promptbreeder are standard in evolutionary algorithms, their application within the domain of LLMs is novel. Also some operators simply cannot be implemented without LLMs, e.g. the lineage based operator, or the context to prompt operator. Designing these operators for LLMs posed unique challenges, particularly in balancing diversity and relevance of the generated prompts. We demonstrate the effective use of LLMs in evolutionary algorithms, a contribution we believe is significant in this field. Furthermore, we argue that the simplicity of the individual mutation operators of Promptbreeder is in fact a strength of the approach, demonstrating that existing mutation operators can be orchestrated to create a capable self-referential self-improving system.

Reviewer comment (Experiments):
Overall, my main question is how such method will compare with prompt tunning method?

Author response:
**1. Comparison with Gradient-Based Prompt Tuning (i.e. Soft Prompting) Methods:**

While we agree that a comparison to soft-prompting methods could be valuable, it falls outside of the scope of this work as well as outside of the scope of prior published state-of-the-art prompt strategies such as Chain-of-Thought Prompting [1], Plan-and-Solve Prompting [2], as well as APE [3] and OPRO [4], to name just a few.

[1] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., … Zhou, D. (2023). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv. https://doi.org/10.48550/arXiv.2201.11903

[2] Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023). Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. arXiv. https://doi.org/10.48550/arXiv.2305.04091

[3] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2023). Large Language Models Are Human-Level Prompt Engineers. arXiv. https://doi.org/10.48550/arXiv.2211.01910

[4] Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., & Chen, X. (2023). Large Language Models as Optimizers. arXiv. https://doi.org/10.48550/arXiv.2309.03409