Paper: Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (ICLR-2024)

Reviewer comment (Reproducibility):
The paper could benefit from more detailed explanations of the implementation details for each type of EA (GA and DE) used in EVOPROMPT. Providing more information on the specific steps and instructions for each algorithm would help readers better understand the proposed method and its connection to LLMs.

Author response:
**A**: Thanks for reminding us of the specific steps, which is helpful for readers to better comprehend our methods. In our original paper, we unify the evolutionary algorithms in a framework. Considering your comments, we have updated our draft and added detailed algorithms for GA and DE, respectively (See Appendix A for details of implementation). We hope this can be responsive to your concerns. Additionally, we have attached the corresponding code in the supplementary materials. Later we will release our code to the public.

Reviewer comment (Experiments):
The paper could include more ablation studies to better understand the impact of different components of the proposed method. For example, analyzing the effect of different population sizes, mutation rates, or selection strategies on the performance of EVOPROMPT would provide valuable insights into the method's robustness and generalizability.

Author response:
**A:** Sec. 5.1 and 5.2 in our original paper have explored the effect of important components designed in GA and DE respectively. Particularly, for GA (Sec. 5.1), we mainly analyze the selection strategy including the random, tournament roulette wheel strategy. For DE (Sec. 5.2), we demonstrate the importance of mutation over different parts and the selection of Prompt 3 in Figure 2. Due to the space limitation, the study on population sizes is in Appendix C.1 of our paper.

Besides, we would like to appreciate your suggestion about the mutation rates. Considering the LLMs may struggle with the rate of concrete float value since at the level of natural language, it is hard to measure the mutation extent, we leave this for future work.

Reviewer comment (Reproducibility):
Could you provide more detailed explanations of the implementation details for each type of EA (GA and DE) used in EVOPROMPT? Specifically, it would be helpful to understand how the algorithms are adapted to work with discrete prompts and how the LLMs are used to generate new candidate prompts.

Author response:
**A**: LLMs play a role in generating new prompts based on evolutionary operators including mutation and crossover according to the given candidates. Figure 1 and Figure 2 in the paper provide detailed instructions on guiding LLMs to generate a new discrete prompt. Meanwhile, we utilize the selection and updating strategy in evolutionary algorithms to control convergence and effectiveness.

Reviewer comment (Experiments):
How is the effect of different population sizes, mutation rates, or selection strategies on the performance of EVOPROMPT

Author response:
**A:** Sec. 5.1 and 5.2 in our original paper have explored the effect of important components designed in GA and DE respectively. Particularly, for GA (Sec. 5.1), we mainly analyze the selection strategy including the random, tournament roulette wheel strategy. For DE (Sec. 5.2), we demonstrate the importance of mutation over different parts and the selection of Prompt 3 in Figure 2. Due to the space limitation, the study on population sizes is in Appendix C.1 of our paper.

Besides, we would like to appreciate your suggestion about the mutation rates. Considering the LLMs may struggle with the rate of concrete float value since at the level of natural language, it is hard to measure the mutation extent, we leave this for future work.

Reviewer comment (Experiments):
EvoPrompt's performance appears to be influenced by the quality of the initial prompts (initial population), which may limit its applications. While the paper emphasized the DE version's superior exploration over the GA, a deeper analysis would be beneficial.

Author response:
> Q: EvoPrompt’s performance may be influenced by initial prompts.

A: In Sec 5.3, we investigate the effect of initial population quality. The results of random-k and top-k initialization exhibit a high degree of similarity and the random setting is more in line with the real situation. This suggests that EvoPrompt does not depend on carefully crafted manual prompts. Instead, even randomly selected prompts can achieve a good performance. Additionally, though DE (48.64) and GA (47.80) starting from the selected bottom population perform not that well, they still outperform APE (46.32), demonstrating the effectiveness and robustness of EvoPrompt.

Reviewer comment (Evaluation):
Further discussion and evaluation of the diversity of the prompts produced by EvoPrompt would be beneficial.

Author response:
> Q: Deeper analysis for DE’s exploration over GA. \& Diversity of prompts

A: We are very grateful for your insightful comments on comparing DE and GA from this perspective. Considering your suggestions, We mainly analyze prompt diversity through the following aspects: 1) the average length of the prompt and the corresponding variance in the population; 2) the number of new words after each iterative step; 3) the case study. We will briefly summarize the results in this response. 

* **Average length and variance**: From Figure 9 (a, b) in the Appendix, DE shows higher length and variance than GA. This implies that the prompts generated in DE are more diverse in terms of length. 

* **Number of new words**: From Figure 9 (c), we can observe that in the latter iterations, DE mutates more new words than GA, and thus shows better potential to explore and escape from the local optimum.

* **Case study**: in Figure 1 (GA) and Figure 2 (DE), the given examples are taken from our experimental logs. 
**1)** For GA, in the final prompt (at the bottom), words in orange and blue are inherited from the parental prompts respectively, and words in black are mutated parts. 
**2)** For DE, comparing the final generated prompt (wrapped by a box at the bottom) with the basic prompt (in the upper part of the figure, prompt in blue words), the new prompt keeps half of the original one, while the other half derives from crossover and mutation referring to other candidates. DE keeps the common parts between the given candidates as repetition and only performs mutation on different parts. Both of them have repetitions and mutations, corresponding to exploitation and exploration.

Results related to prompt diversity revealed that DE tends to generate more diverse prompts than GA. This result also indicates that DE is potentially better at exploration than GA. Please see Appendix C.4 in our revised version for more details. Thank you again for your insightful comments on comparing DE and GA from this perspective.

Reviewer comment (Experiments):
The EA optimization for a single prompt involves iterations. However, the search doesn't appear to converge in Figure 6. Is it possible to illustrate the effects of more iterations? It's unclear if these 10 iterations are conducted within a single LLM session or if each iteration starts with a new session. Besides, more details should be provided regarding the runtime, and resources for running the GA.

Author response:
> Q: The search doesn't appear to converge and it is possible to illustrate the effects of more iterations.

**A**: We increase the iteration steps and there’s still slight performance gain on the Subj dataset, but EvoPrompt, both GA and DE versions, on most of the datasets converge around 10 steps. To maintain uniformity and reduce hyperparameter elaboration in the experimental setup, we selected a consistent value of 10 across all tasks. 
It's true that at the iteration of 10, the average score still shows an increasing trend but there exists a trade-off between the number of iterative steps and performance. If we continue evolving until convergence, the best score of DE can rise. 
Please refer to Appendix C.3 for more detailed demonstrations.


> Q: Are 10 iterations conducted in a single LLM session or a distinct new session?

A: Each evolution is in a distinct new session.

> Q: Details about the runtime and resources.

A: Please refer to our general responses on cost analysis for details related to resources. The runtime varies from each dataset, the table below lists the runtime when optimizing for Alpaca on a single V100 GPU (except that BBH is optimized for GPT3.5). Additionally, the runtime of evaluation depends on iterations, the size of the population and the size of the dev set. Here we show the time under our default setting. The time for evaluation is almost the same for APE and our methods. Note that each task in BBH takes about the same amount of time to run.

|                | SST-2 | CR   | MR   | SST-5 | AG's News | TREC | Subj | ASSET | BBH  |
| -------------- | ----- | ---- | ---- | ----- | --------- | ---- | ---- | ----- | ---- |
| Runtime (mins) | 63    | 63   | 65   | 90    | 162       | 99   | 194  | 163   | 86   |

Reviewer comment (Evaluation):
It would be beneficial if the authors could present negative examples where EvoPrompt failed, as this could inspire direction for future research. Moreover, the limitations of this approach should be discussed.

Author response:
> Q: Negative examples where EvoPrompt fails and the limitations can lead to future investigation.

A: In our results of Big-Bench Hard, there are several tasks that either EvoPrompt (GA) or EvoPrompt (DE) fails. This is because the distribution of the randomly sampled dev set is not aligned with the test set, resulting in a performance gap between the evaluations based on them. Accordingly, though EvoPrompt gets a better prompt based on the dev set, it may not perform as well when tested on the test set. Thus, constructing a more effective dev set could further enhance the performance, which needs future investigation.

Reviewer comment (Experiments):
In EvoPrompt, most of the initial populations were generated with a mix of manually-created and LLM-generated prompts. What is their ratio when DE works better than GA? What would be the performance of EvoPrompt if the initial prompts were all generated by LLM?

Author response:
> Q: What is the ratio of manually created and LLM-generated prompts in the initial population when DE works better than GA? & What would be the performance of EvoPrompt if the initial prompts were all generated by LLM?

A: There are two ratios of manually written prompts in the initial population, before top-k and after top-k and variations. Before selecting top-k, the ratio of manually written prompts is around 50\%. The ratio after top-k and variations are listed in the table below. We haven't observed a significant correlation between the ratio of manual prompts in the initial population and the performance. 

|                 | SST-2 | CR    | MR    | SST-5 | AG's News | TREC  | Subj  | ASSET |
| --------------- | ----- | ----- | ----- | ----- | --------- | ----- | ----- | ----- |
| Score (GA)      | 95.13 | 91.27 | 90.07 | 49.91 | 72.81     | 64.00 | 70.55 | 46.43 |
| Score (DE)      | 94.75 | 91.40 | 90.22 | 49.89 | 73.82     | 63.73 | 75.55 | 46.21 |
| Ratio of manual | 0.37  | 0.50  | 0.20  | 0.23  | 0.47      | 0.00  | 0.30  | 0.33  |

Additionally, here we conduct experiments with the initial prompts generated by LLMs completely, as shown in the table below. We can observe that for SST-5, initialization has little impact. 

| Method             | SST-5 |
| ------------------ | ----- |
| APE                | 46.32 |
| GA                 | 49.91 |
| GA (-manual)| 49.98 |
| DE                 | 49.89 |
| DE (-manual) | 50.06 |

Reviewer comment (Experiments):
Will there be any benefits to combining and alternating GA and DE updates during search?

Author response:
> Q: Will there be any benefits to combining and alternating GA and DE updates during the search?

A: Thanks for your comments. It’s a promising suggestion. The selection and updating strategy differ from DE and GA, making the combination non-trivial. We will further investigate this for our extensions.

Reviewer comment (Novelty):
While I really appreciate the evo operator that leverages LLMs, I have more doubts about the relevance of the considered application. I feel like there are more interesting applications for this method than prompt engineering which I believe will soon not be a thing anymore as we see the latest language models becoming more and more robust to the prompt that is used. However, I acknowledge that there are ongoing efforts in that field and while I personally don't find it very exciting or relevant, this work contributes efficiently to this research track.

Author response:
Firstly, we would like to clarify our **motivation for focusing on prompts**. 

* We kindly argue that prompts for large language models (LLMs) are crucial, as provide a necessary interface to interact with black-box LLMs. In the era of LLMs, the fine-tuning paradigm is computationally demanding, while the prompt in natural language is effective, interpretable and human-readable, especially practical for those limited by the computational budget. Previous works find prompt order [4], templates and examples can highly affect [5] the performance. 
The field of prompt engineering has recently gained significant popularity, evolving into a highly sought-after career. Automating this process, and achieving superior performance compared to human-designed prompts, signifies a major step forward in the area of language model optimization. This motivates us to study the automated generation of high-quality prompts. 

* We understand your concern that prompt engineering might not be useful as the language model scales. However, it has not been proved that the larger the model is, the more robust it is to the given prompt. In our experiments, even GPT3.5 is sensitive to prompts. Lu et al. [4] demonstrate that models of different scales are sensitive to the prompt, with the largest LM being 175 billion. Meanwhile, our proposed EvoPrompt also achieves significant improvement on multiple tasks (e.g., up to 25% improvement on BBH compared with the famous Chain-of-Thought, which shows the influence of different prompts). Moreover, as the model's scale increases, the corresponding rise in computational cost renders fine-tuning methodologies less feasible. This shift amplifies the importance and relevance of prompts, known for their effectiveness and better interpretability.

* One major benefit of employing LLMs as evolutionary operators is their ability to take into account the coherence within a sequence. Consequently, the prompt in natural language is a particularly appropriate application. We begin with NLP tasks because they come with standard objective evaluation criteria, which are more effective in showcasing the performance of EvoPrompt. As you mentioned, it would indeed be intriguing to apply EvoPrompt to multi-modality applications (e.g., game level and image generation) by adjusting the evaluation and selection strategies.

Reviewer comment (Novelty):
I also have some doubts about the relevance of this work to ICLR. In my opinion, the main contribution of this work is this LLM-based mutation/crossover operator which would make more sense in an evolutionary algorithms conference such as GECCO than at ICLR.

Author response:
Secondly, **ICLR considers a broad range of subject areas.** One contribution of our work is to connect LLMs with evolutionary algorithms, where, as you mentioned, the most innovative design is the LLM-based mutation/crossover operator. 
This connection does not contribute to improving the evolutionary algorithms, instead, it sparks new applications for both of these fields. Accordingly, besides evolutionary algorithms, it is also highly related to LLMs. For example, the works considering evolutionary algorithms with reinforcement learning [2, 3]  were also accepted by ICLR’22 and ICLR’23. Another contribution is the prompt optimization framework. While our main focus is on prompts for NLP tasks, potential applications could also include prompts for vision tasks. This is in line with the "applications in language and vision" in the call for papers, as well as other previously accepted papers at ICLR'23 like APE [1].

Reviewer comment (Evaluation):
I think that this work would strongly benefit from a study of the computational cost of the approach in the main core of the paper. I saw some elements that could help towards this end in the supplementary, but I would like to see an analysis of the computational cost / financial cost and time needed to perform this optimization. It would be interesting also to add similar analysis for the baselines. I think this would be very useful to the readers to evaluate to what extent they could apply that to their research if they have either to serve themselves Alpaca on an accelerator or to query the OpenAI API after connecting their credit card.

Author response:
Finally, to address your **criticism regarding costs**, we analyze the cost to apply our EvoPrompt in general responses. Intuitively, there exists a trade-off between costs and the corresponding performance. EvoPrompt shows a higher ceiling compared with APE. Besides, EvoPrompt outperforms APE when keeping the same number of API requests. Please see the general response for more statistics.

Reviewer comment (Novelty):
The authors mention that EA are good at balancing exploration and exploration, however it is not always the case and this is for instance often a limitation of GA methods. I would recommend the authors to have a glance at the Quality Diversity literature with algorithms such as MAP-Elites which are much better at balancing exploration. I think it would be very exciting to use EvoPrompt on top of a QD method.

Author response:
* **QD methods**: Thanks for this insightful suggestion and for reminding us of the limitation of SA compared with EAs. QD methods like MAP-Elites are more recent and seek to find a set of policies with both high rewards and diverse behaviors. These approaches show a promising ability to balance exploitation and exploration, which is crucial for heuristic searches. We will further investigate this for future work.

Reviewer comment (Experiments):
In the conclusion, the authors mention looking at Simulated Annealing (SA) for future work. In my experience, GA and other more recent EAs always outperform SA by a strong margin (if correctly tuned and implemented). Thus, I would not advise going in that direction. QD methods are probably going to be more fun. Looking at multi-objectives methods such as NSGA-II might be interesting as well.

Author response:
* **QD methods**: Thanks for this insightful suggestion and for reminding us of the limitation of SA compared with EAs. QD methods like MAP-Elites are more recent and seek to find a set of policies with both high rewards and diverse behaviors. These approaches show a promising ability to balance exploitation and exploration, which is crucial for heuristic searches. We will further investigate this for future work.