Paper: Prompt Engineering a Prompt Engineer (ICLR-2024)

Reviewer comment (Reproducibility):
The clarity of the way to update hard prompt with gradient-based optimizer can be enhanced and improved by providing details, especially how do we access the gradients to help LLM refine hard prompts?

Author response:
* As mentioned in the example above, our method is not a gradient-based optimizer. The textual feedback generated by the model is used to steer the prompt editing process, and therefore is analogous to “gradients.” We hope the example above helps clarify this.

Reviewer comment (Experiments):
The iteration for the optimizer is set to 3 in the experimental setup. How does this come from and be enough to optimize prompts? This part is not well-supported.

Author response:
* We set the number of prompt edits to be 3 due to cost concerns. Empirically, performance tends to plateau after 3 steps.  
For each prompt candidate, we need to run it on the training set to find the model errors and run it on the dev set for validation and prompt selection. This leads to a significant number of LLM calls.

Reviewer comment (Experiments):
In Figure 3 and Figure 4, I am not sure why are we doing comparison across different training timestamps. Instead, should we focus on the final timestamp or the converged step to confirm it's finalized and optimized for any further investigations? This comparison to display the dynamics is confusing me.

Author response:
* We agree with you that the prompt on final/best prompts are most important. Our table 1 and table 2 are dedicated to results on the final prompt. (Now they are table 3 and table 1 in the updated paper)  
* Figure 3 and 4 illustrates the _accuracy distribution_ of the proposed prompts over the course of prompt engineering. We create them for some additional analysis and they indeed provide valuable insights. For example, in Figure 3 (now Figure 4 after paper update) we conducted an ablation study on the meta-prompt components and we found that by removing these components, prompt quality at each timestamp is lowered.

Reviewer comment (Writing):
The footnote #3 in page 4 is not well-supported and is confusing. How does the analogy come from and be translated?

Author response:
* Please refer to our clarification section above. We believe it is helpful in answering this question.  
* The textual feedback generated by inspecting a batch of model errors, and therefore this is analogous to “loss.backward()”.  
* Generating a new prompt based on the old prompt and the textual feedback, is analogous to “optimizer.step()”.

Reviewer comment (Writing):
In paragraph "Incorporating Concepts in Optimizers", what's the concept here? Is there any formal definition about concept?

Author response:
* The concepts are batch size, step size and momentum that are commonly used in gradient-based optimization. The bulletin points in that paragraph provide more detail on each of them.  
* Given feedback from reviewer x9xJ and reviewer uHAL, we agree this is a little confusing and we decide to more this part to the appendix in our next paper update.

Reviewer comment (Evaluation):
(minor comment) In Figure 1, the results can be improved by adding variance/range in the accuracy performance. It helps to enhance the soundness of this work.

Author response:
* Thank you for raising this! We will do this whenever applicable. For bars presenting the average across multiple tasks, there is not a straightforward way to compute variance and thus we cannot include them here.

Reviewer comment (Experiments):
The evaluation uses Text-Davinci-003 as the main model but GPT4 when handling the meta-prompt, prompt evaluation and update proposals. It's noted that for two baselines that used the Text-Davinci-002, the results are recreated using Text-Davinci-003. It begs the question of whether APO or APE with GPT4 paraphrases could potentially perform better than reported. Was GPT4 also used in any of the baselines? Have the authors evaluated using Text-Davinci-003 to also handle the meta-prompt? Given that hallucinations hurt performance considerably it would be important to answer these questions for a fair comparison to baselines.

Author response:
* Yes. Our experiments on APO and Iterative APE are using GPT-4 as the prompt proposal model. We make sure the experiments are controlled and __the meta-prompt is the only changing variable__.
* In the initial stage of this project we tried using GPT-3.5-turbo as the prompt proposal model. The overall impression we have is that GPT-3.5-turbo is sometimes confused at the concept of prompt (task description) and the task input and cannot distinguish them. So we decide to proceed with GPT-4

Reviewer comment (Experiments):
Text-Davinci-003 further lacks some capabilities of GPT3.5 and GPT4. Although I am sympathetic to the cost constraints, I would find it valuable if the authors had any existing experimental results where either of these models is used as the main model to optimize for (even if the results are not fully run across all benchmarks).

Author response:
* We decide to use instruct models to be consistent with prior work. Unfortunately we didn’t use GPT 3.5/4 models as the task mode because they are chat completion models.

Reviewer comment (Evaluation):
I suspect that the authors will agree that as a result, one could argue that the meta-prompt itself could be promptly optimized with PE2. Although they allude to this at the very end, I believe that a more detailed discussion on this front could be useful.

Author response:
* Yes, we are very excited about this potential future direction!! 
* Conceptually, we can replace $p^{(t)}$ and $p^{(t+1)}$ in Eq (2) with the meta-prompt $p^{meta}$ directly. However, we think three challenges (and broader questions) arise if we pursue this method:
  * How to collect data for such a study? To ensure this meta-prompt is universal we may need a large collection of tasks along with prompt optimization history associated with them. Creating a resource like this will be a large effort.
  * How to automatically optimize the meta-prompt when there are no ground truth labels for prompt engineering? Math problems have ground-truth answers so that PE2 can reason on them. The task of prompt engineering does not have ground truth labels, and this makes the meta-prompt optimization process more fuzzy.
  * It would be very costly to run and even evaluate a system like this. To evaluate one $p^{meta}$ candidate, we will need to use it for prompt optimization on various tasks as evaluation. We would expect the “learning” process of the meta-prompt to be a magnitude more costly.
* We noticed two really exciting recent works related to this direction archived in the past two months: https://arxiv.org/abs/2309.16797 and https://arxiv.org/abs/2310.02304

Reviewer comment (Novelty):
The algorithmic contribution seems a bit thin. While this might be expected given the black box trial-and-error nature of prompt engineering, there is very little by way of novelty beyond the meta prompt design itself. The issue of limited novelty might be alleviated with additional insight about algorithmic components. For example, why does performance plateau quickly as $t$ increases? What kinds of prompts are generated at the end of long searches?

Author response:
* In response to “Why does performance plateau quickly as t increases?”: Our hypothesis is that “let’s think step by step” is already a near-optimal prompt and thus the performance plateaus quickly when using it as the initialization. According to Figure 4(b) in the paper (now Figure 3(b) in the updated paper), when alternative initializations are used, PE2 is able to improve the performance gradually and the performance hasn’t plateaued at t=3.
* In response to “What kinds of prompts are generated at the end of long searches?”: We summarized some notable prompt edits in Table 4, 5, 8 (Now Table 4, 5, 9 in the updated version). PE2 is able to correct wrong or incomplete task instructions, provide more specific context and details. In our upcoming paper update, we will show that PE2 can lay out multi-step plans in the prompt.

Reviewer comment (Presentation):
The current presentation makes it hard to "separate the wheat from the chaff". I found it challenging to quickly identify the "final" best variant of the meta-prompt. The paper describes a number of components, but discards some in the final version constituting PE2 (used to generate Figure 1). If I've understood correctly, discarded items include the prompt tutorial, step size and momentum. If true, perhaps the main paper might be simplified to only describe what is actually used in PE2 / Figure 1 with supporting evidence with the rest moved to the appendix as negative experiments. Just a suggestion.

Author response:
* Thank you for raising this point! This is also pointed out by reviewer x9xJ and thus we will update our paper by moving the meta-prompt components with mixed observations to the appendix. We will push an update to the paper very soon.

Reviewer comment (Reproducibility):
How exactly are the examples in Line 42 in B.4 selected? It seems like PE2 uses 2 negative examples from D_train ("hard negative sampling"). Is this correct? Does batch_size refer to the total number of examples or just the negative examples?

Author response:
* Yes, by default we _only_ select examples in D_train that the model produces _wrong_ answers. In this case batch_size = # negative examples = # selected examples.
* We tried selecting random examples in D_train instead of negative examples, and this is denoted as “- hard negative sampling” in Table 1 (now Table 3 in the updated paper). Performance is slightly worse when using random examples.

Reviewer comment (Experiments):
Appendix C.1.1 suggests including more in-context examples (e.g., 3, 10) improves performance. If so, why is batch size set to 2 in PE2?

Author response:
* Apologies for the confusion! After the main experiments on academic benchmarks are done, we decide to try out our method on an industrial prompt. The industrial prompt is substantially longer (5k+ tokens), which creates a very different optimization space. The default hyperparameters on academic benchmarks do not work directly. This necessitates a separate hyperparameter study, which we described in Appendix C.  Furthermore, to the best of our knowledge this is the first work to look at directly optimizing very large prompts deployed application prompts.

Reviewer comment (Writing):
Is the use of the optimization terminology beneficial? The term "batch" in the context of LLMs is reasonably well understood to refer to the set of examples used during fine-tuning and much less so to refer to the set of input-output examples in the prompt. Since step size and momentum don't seem to be anyway used in the final PE2 version, might it be clearer to simply describe the meta-prompt in its final form without reusing popular, well-understood terminology from optimization?

Author response:
* As mentioned earlier, after reading the reviews we realized that this causes confusion and we will push an update to the paper. Thank you for the suggestion!

Reviewer comment (Evaluation):
The effectiveness of the proposed approach is not conclusive. Although the approach includes a few interesting tricks, the effectiveness of them is unclear, as indicated by the ablation study. "The optimizer-inspired concepts can improve the performance occasionally, but the current experiments do not give a definitive conclusion regarding their utilities"; "We do not observe significant improvement by incorporating prompt engineering tutorial." While the readers appreciate the author's honesty and agree negative results are still informative, it would be good if the author explores more on which scenarios the proposed tricks are more likely to be helpful.

Author response:
* Sorry about the confusion! To clarify, our ablation study shows that __three components we propose (two-step task instruction, context specification and step-by-step reasoning template) are indeed effective__ (Table 1 and Figure 3)(They are now Table 3 and Figure 4 in the updated paper). We believe these are important contributions in our paper. PE2 powered by these components also outperform prior methods such as Iterative APE and APO on a wide range of tasks (Figure 1).
* We agree that results on optimizer-inspired components such as optimization history and step size are currently mixed. As pointed out by reviewer uHAL, we are re-organizing the paper to highlight the effective components and defer the remaining mixed results in the appendix. Please expect an updated version of the paper before the discussion period ends.

Reviewer comment (Experiments):
For GSM8K, the SOTA is above 90, while the authors use less strong foundation models. The big gap between the SOTA and the results obtained still raises concerns about the effectiveness of the methods. Does it work on better foundation models?

Author response:
* Our experiments on GSM8K is about finding the optimal prompt in the __zero-shot chain-of-thought__ setting, as done in https://arxiv.org/abs/2205.11916.
* We learned that the current state-of-the-art performance on GSM8K is based on __few-shot chain-of-thought prompting__ or __additional human-designed verification workflow__. We believe these advances are orthogonal to us and the results are not directly comparable.
* While we agree pushing forward state-of-the-art performance is important, we would like to highlight that __our focus is enabling LLMs to improve LLM performance automatically__. We really hope one day LLMs can conduct research and push forward state-of-the-art by themselves one day (https://arxiv.org/abs/2310.03302).

Reviewer comment (Experiments):
Some recent papers related to prompt design are also interesting. How do the proposed methods compare with these methods? Examples include meta prompt optimization (section 4.2 of https://arxiv.org/pdf/2309.03409.pdf) and prompt design to optimize demonstrations (https://arxiv.org/abs/2305.14106).

Author response:
Thank you very much for bringing these works to our attention!

Regarding OPRO (https://arxiv.org/abs/2309.03409):
* Firstly, please kindly note that OPRO was archived three weeks before ICLR submission deadline. We request that the novelty and contribution of our work is evaluated independently.
* In our submitted version, we discussed our differences and thoughts on OPRO in Appendix A.1. We think OPRO and PE2 represent two very different optimization strategies. OPRO mainly uses prompt accuracy as reward signal and asks for _implicit_ optimization; PE2 mainly uses model failure cases for generating _explicit_ feedback.
* As mentioned in the OPRO paper, “We consider incorporating explicit natural language feedback on generated solutions for later optimization steps as _future work_.” PE2 can be considered as initial steps towards this direction.

Regarding Self-Adaptive Prompting (https://arxiv.org/abs/2305.14106):
* Thank you for bringing this work to our attention! It’s a cool idea to automatically construct in-context demonstrations based on consistency, diversity and repetition. We will include it in the related work section in our upcoming update.
* In our study we limit our scope to prompt engineering the task descriptions (see our footnote 1). Following prior work (https://arxiv.org/abs/2205.11916), the prompts are limited to having less than 50 tokens in our experiments. Hence the two problems settings are not consistent and we cannot make direct comparison.
* It will be interesting to study the synergistic effect of self-adaptive prompting and our PE2 method, which we leave as future work.