Paper: Large Language Models as Optimizers (ICLR-2024)

Reviewer comment (Novelty):
What is the key/unique advantage of using LLMs to optimize over some traditional optimization algorithms, especially on classical optimization problems (not prompt engineering)?

Author response:
The key/unique advantage is that one can use natural language to describe the optimization problem, as discussed in the second paragraph of Section 1: “​​instead of formally defining the optimization problem and deriving the update step with a programmed solver”. This can make optimization more accessible to general users who do not have much domain knowledge of the types of optimization tasks in question, and may also boost productivity of optimization experts who work on these tasks on a daily basis.

Meanwhile, we would like to clarify that the focus of our work is not to solve all classic optimization problems. We evaluated our approach on linear regression and Traveling Salesman Problems only as motivating examples to demonstrate the potential of LLMs for optimization. Instead, our main task is prompt optimization which is a known challenge for classic optimization methods, where the optimization space is in natural language.

Reviewer comment (Experiments):
For prompt optimization, the optimization process of this work (i.e., directly feeding solution-score pairs, optimization task descriptions, and meta-instructions into the optimizer LLM) is a black box/lacks interpretation; why is it a better prompt optimizer than those new methods that leverage LLMs to explicitly act as mutation and crossover operators, and further optimize the prompt? Such as:  
1. EvoPrompting: Language Models for Code-Level Neural Architecture Search by Chen et al.  
2. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers by Guo et al.

Author response:
Thanks for pointing out these related works. We have discussed such evolutionary methods in Section 6. Following your suggestion, we also added an empirical comparison with EvoPrompt. **Please refer to our general response for a summary, and we summarized the full results in Appendix M of the updated Supplementary Material.**

Reviewer comment (Experiments):
To further understand the effect of the purple text in this work, an ablation study may be beneficial for improving the solidness of the results.

Author response:
We have benchmarked the effect of exemplars (i.e., the purple text) in ablation studies. Please refer to Figure A17 in Appendix K of the Supplementary Material. We show that adding exemplars boosts performance, since it offers more context to the optimizer LLM and enables it to generate more relevant instructions (prompts). For example, on BBH tasks, the optimizer LLM found instructions that are tailored to the tasks, as shown in Table A4 and A5 in Supplementary Material. Without exemplars in the meta-prompt, it is impossible for the optimizer LLM to understand the task if the initial prompts are simple and generic.

Reviewer comment (Experiments):
Numbers of examplers. Did you take the randomness of example picking into consideration? For each run of every setting, do you give the same set of examples?

Author response:
At each step, we randomly sample a different set of 3 exemplars to put into the meta-prompt.

- How we chose 3 for the number of exemplars: Our ablation (Figure A17(c) and A17(d) in Appendix K of the Supplementary Material) shows that 3 achieves a good balance between the informativeness of the meta-prompt and the context length constraint. 
- How randomness is useful: We randomly sample exemplars at each step because it reduces the risk of overfitting on a fixed set of exemplars, and reduces the chance of the optimization getting stuck. The latter is because if both the set of exemplars and the top instructions in the meta-prompt remain the same (this more often occurs in later steps when the performance starts to saturate), the meta-prompt will remain the same, and the optimizer LLM may keep generating the same instructions.

Reviewer comment (Experiments):
I noticed that for different tasks, the “batch size” that works the best can be different (Figure 5, cd). Do you find any obvious patterns on which types of data/tasks prefer a smaller “batch size” and vice versa?

Author response:
Insightful question. Although setting the batch size of 8 generally achieves the best final performance across models and tasks, some other batch sizes may work equally well in certain cases (like batch size 1 and 2 at the early stage of optimization in Figure 5(d)). In general, the batch size used by our approach can be smaller for easier optimization problems, such as:
- When the scorer LLM is instruction-tuned, the accuracy landscape will be less bumpy because the model performance is less sensitive to the prompt, thus a smaller batch size would suffice for the accuracy curve to trend up. The bumpiness can be well reflected by the oscillation in the optimization curve: both on the same GSM8K 3.5% training set, the curve on the pre-trained PaLM 2-L scorer (Figure 1(a)) is much more bumpy than the blue dots in Figure 5(a), especially at early steps. This means OPRO on a pre-trained scorer LLM needs a larger batch size (like 8) to succeed. 
- When the task is easier, one doesn’t need to carefully prompt the model to get a good score, meaning the loss landscape is also less bumpy. One example is the difference between Figure 5(c) and 5(d) in the main paper: BBH sports_understanding is easier than GSM8K in that the same text-bison scorer LLM achieves around 30% higher accuracy, close to 100% on sports_understanding, thus a smaller batch size is enough.

Reviewer comment (Evaluation):
The biggest limitation is that OPRO's performance looks highly fluctuating. It's unclear if the LLM really finds the so-called optimization "trajectory" or just randomly finds a good prompt. The authors should provide more analysis to show that the LLM is indeed learning to optimize.

Author response:
We acknowledge that the optimization curves do not show monotonic increase at every step. Most optimizers don’t, especially when the loss landscape has a more complicated curvature. From our new experiment showing that the one-step prompt generation method you suggested performs much worse than our approach, we observe that LLM gradually improves upon previous prompts based on the optimization trajectory, just like what other optimizers do.

In addition, in figures showing the accuracy at each step (Figure 1 in main paper; Figures A11, A12, A13 in Supplementary Material), we presented the *average* (instead of the maximum) accuracy of 8 instructions generated at each step, which showed generally upward trends. This shows the optimizer LLM gradually generates *distributionally* better instructions. Furthermore, Figure 5(c) and 5(d) showed that with the same number of total evaluated instructions, generating 8 instructions per step (default setting) achieved better performance than generating 16 instructions per step, demonstrating that the LLM learns to optimize prompts from previous attempts better with more iterations of update, instead of just randomly finding a good prompt by sampling more instructions.

Reviewer comment (Experiments):
Limited exploration on how to provide richer feedback to LLM beyond aggregated scores. It could help address limitations.

Author response:
As noted in the discussion of future work in Section 7 (Conclusion), “we tried including error cases in the meta-prompt rather than randomly sampling from the training set at each optimization step, but the results are similar, indicating that the error cases alone are not informative enough for the optimizer LLM to grasp the cause of the wrong prediction”. As also noted in Section 7, we agree that improving our approach to better incorporate more detailed feedback is a promising direction for future work. 

That said, we show that our simple meta-prompt still demonstrates significant improvement across different tasks and LLMs, and we see simplicity as a key advantage of our method.

Reviewer comment (Experiments):
No comparison to other prompt optimization methods. It could better situate contributions.

Author response:
Thanks for your suggestion. We added an empirical comparison to EvoPrompt, which is an evolutionary method pointed out by Reviewer iHbx. **Please refer to our general response for a summary, and we summarized the full results in Appendix M of the updated Supplementary Material.**

Reviewer comment (Evaluation):
Limited analysis. For example, there is no characterization of what makes an effective prompt.

Author response:
We have studied both aspects in the ablation studies. Please see Section 5.3 in the main paper and Appendix K in the Supplementary Material for ablations on how different parts of the meta-prompt and temperature matter.

Reviewer comment (Reproducibility):
Can you clearly state how many optimization steps are performed in each experiment? How does the number of steps affect performance?

Author response:
We run a maximum of 200 steps in every experiment although many converge much sooner. The training accuracies trend upwards until plateaued, so the performance generally improves with more number of steps. As shown in Figure 1 in the main paper and Figures A12, A13 in the Supplementary Material, the performance often plateaus within 100 steps.

Reviewer comment (Experiments):
Can you provide an experiment where you generate the same number of prompts in one step as your current experiments, and evaluate them all, and report the best one? This would help clarify whether the LLM is really learning to optimize or just randomly finding a good prompt.

Author response:
Sure, based on your suggestion, we conducted the following additional experiment on GSM8K and BBH sports_understanding with the PaLM 2-L-IT optimizer. For GSM8K the scorer LLM is pre-trained PaLM 2-L and the initial instruction (i.e. prompt) is “Let’s solve the problem”, and for BBH sports_understanding the scorer LLM is text-bison and the initial instruction is the empty string. We generate 50 instructions (i.e. prompts) in only one step, thus the meta-prompt only includes task exemplars, the performance of the empty string, and some meta-instructions. All the other hyperparameters are the same as our current experiments in paper.

Our results show that this one-step prompt generation performs much worse than our approach. Specifically:

- On GSM8K, the best instruction among all 50 is still “Let’s solve the problem”, with a 64.4 training accuracy and a 60.8 test accuracy. On the other hand, our approach (corresponding to Figure 1(a) in the main paper) found “Let’s do the math!”  with a 78.2 training accuracy and a 76.3 test accuracy at the 5th step by generating 8 instructions at each step.
- Similarly, on BBH sports_understanding, the best instruction among all 50 achieved a 84.0 training accuracy and 80.0 test accuracy. This is again worse than the instruction found by our approach at Step 4, which achieved a 88.0 training accuracy and a 84.5 test accuracy.

**We have also added the above results to a new paragraph “Comparison with one-step instruction generation” in Appendix K of the Supplementary Material.**

Reviewer comment (Experiments):
For prompt optimization, have you experimented with providing more detailed feedback to the LLM beyond aggregated scores? (e.g. accuracy on different example types, common mistakes)

Author response:
As noted in the discussion of future work in Section 7 (Conclusion), “we tried including error cases in the meta-prompt rather than randomly sampling from the training set at each optimization step, but the results are similar, indicating that the error cases alone are not informative enough for the optimizer LLM to grasp the cause of the wrong prediction”. As also noted in Section 7, we agree that improving our approach to better incorporate more detailed feedback is a promising direction for future work. 

That said, we show that our simple meta-prompt still demonstrates significant improvement across different tasks and LLMs, and we see simplicity as a key advantage of our method.

Reviewer comment (Experiments):
How does the meta-prompt length affect optimization performance? Is there a sweet spot balancing past solutions and problem descriptions?

Author response:
As for the meta-prompt length: we have benchmarked the effect of the number of exemplars in Appendix K of the Supplementary Material, where we show that while including exemplars is crucial for achieving good optimization performance, including 10 exemplars in meta-prompt harms the optimization performance, which might be because the total length of exemplars becomes too long that distracts the attention to other important components such as past solutions.

Regarding the number of past solutions: we observe that the optimization performance is not sensitive to the number of past solutions in the meta-prompt, as long as that number is not too  small (so that the generated new solution may lack diversity) or too large (exceeding the context length limit). So we include a maximum of 20 top solutions from the past by default. **Please refer to the paragraph “Meta-prompt: the number of past solutions” in Appendix K of the updated Supplementary Material.**

Reviewer comment (Experiments):
What determines the choices of sampling temperature? Have you tried adaptive temperature schedules?

Author response:
We have an ablation study on sampling temperature in Appendix K of the Supplementary Material (Figure A19). The temperature cannot be too small (so the generated solutions lack diversity) or too large (so the generated solutions do not incorporate enough information from the meta-prompt). We tried linearly increasing or decreasing the temperature throughout the optimization process, but the performance didn’t improve on either GSM8K or BBH, so we stick to a constant temperature.

Reviewer comment (Experiments):
What are the limitations on problem complexity that OPRO can handle? Analysis of how performance degrades with complexity?

Author response:
OPRO is able to handle all prompt optimization tasks we have tried. For math optimization, we discuss some failure cases at the end of Section 3 in the main paper (for the Traveling Salesman Problem) and in Appendix C of the Supplementary Material. The main takeaways are:

- OPRO may “get stuck at a point that is neither global nor local optimal”. This happens more frequently when the problem’s dimensionality gets higher, because there are more coordinates to balance in between. In fact, OPRO often only updates one dimension in each step (a.k.a. coordinate descent), which naturally fails more often on higher dimensionality problems.
- OPRO is “hard to navigate a bumpy loss landscape”, for example, the Rosenbrock function.
- On the Traveling Salesman Problem, “the performance of OPRO degrades dramatically on problems with larger sizes”.

Reviewer comment (Evaluation):
Can you better characterize what makes an effective prompt for optimization? Any semantic or syntactic patterns?

Author response:
The last paragraph in Section 4 of the main paper details the crucial components of the meta-prompt, and our ablation studies in Section 5.3 of the main paper and Appendix K of the Supplementary Material Section K demonstrate the necessity and design choices of these components.

Reviewer comment (Experiments):
How does OPRO compare to other gradient-free prompt optimization methods? Could be included in experiments.

Author response:
We added an empirical comparison to EvoPrompt [1], an evolutionary method pointed out by Reviewer iHbx. **Please refer to our general response for a summary, and we summarized the full results in Appendix M of the updated Supplementary Material.**

Reviewer comment (Novelty):
I disagree with the authors fundamentally about what optimization means. To me, this work is not optimization but step-by-step inference. To quote Wikipedia for reference, optimization is "the selection of a best element, with regard to some criterion, from some set of available alternatives." One can plausibly consider the process of prompt selection as "optimization", but in order to make a claim on the general area of optimization I would expect results on optimizing a wide range of convex functions and non convex functions as opposed to word problems. The claim on linear regression as an important result is a relevant but very limited result, but it is not included in the main paper.

Author response:
First, we would like to clarify that the focus of our work is not to solve all kinds of classic optimization problems. We evaluated our approach on linear regression and traveling salesman problems only as motivating examples to demonstrate the potential of LLMs for optimization. Instead, our main task is prompt optimization which is a known challenge for classic optimization methods, where the optimization space is in natural language.

Meanwhile, optimization is a very general concept. As classic optimizers such as gradient descent and Adam, our approach proposes better solutions in an iterative manner, step-by-step. Specifically, instead of updating the current solution with a mathematical rule (like in momentum or Adam) or by inferencing a surrogate model (like in Bayesian optimization), we prompt the LLM to get the next-step solution, hence “LLMs as optimizers”. From this perspective, prompt optimization is also an optimization problem: the objective is to maximize the task accuracy within the huge natural language space.

Reviewer comment (Novelty):
Considering the relevance of this approach to step-by-step inference, what is new here compared to previous step-by-step inference procedures? The authors include them in the introduction as evidence for LLM is capable of doing multi-step reasoning, but do not distinguish differences between this work and prior work. Also they do not compare to these prior works.

Author response:
These works are complementary to our work. All 3 works used *manually written prompts* to improve LLMs’ performance, while our method focuses on *how to automatically find such prompts via optimization* and in many cases we can find even better prompts compared to the manually written ones in these existing works. 

More specifically, https://arxiv.org/pdf/2205.11916 shows LLMs can do zero-shot chain-of-thought prompting; https://arxiv.org/abs/2201.11903 shows chain-of-thought prompting improves LLMs’ reasoning performance; https://arxiv.org/abs/2305.10601 shows tree-of-thought improves the performance on several tasks that require planning and search. These papers are about manually improving the LLM performance with manually written prompts, while our approach is on prompt optimization/tuning. For example, our work showed how our method greatly improved zero-shot chain-of-thought prompting.

Reviewer comment (Novelty):
What is the difference between this work and other step-by-step inference techniques such as https://arxiv.org/pdf/2205.11916, https://arxiv.org/abs/2201.11903, https://arxiv.org/abs/2305.10601

Author response:
Again, we would like to clarify that the works you mentioned are all about manually written prompts. Specifically:
- Chain-of-thought prompting paper (https://arxiv.org/abs/2201.11903) repetitively stated that they “manually composed chains of thought to use as few-shot exemplars” in sections on experiments (Section 3-5).
- Zero-shot chain-of-thought prompting paper (https://arxiv.org/pdf/2205.11916) manually designed zero-shot instructions to trigger chain of thought, and show that “Let’s think step by step” achieves the best performance among all their manually written instructions.
- Tree-of-thought prompting paper (https://arxiv.org/abs/2305.10601) manually designed prompts for thought generation and thought evaluation.
- ReAct (https://arxiv.org/pdf/2210.03629.pdf) stated that its prompting scheme “manually compose ReAct-format trajectories to use as few-shot exemplars in the prompts” (e.g., in Section 3.2).
 
In our evaluation on prompt optimization, our focus is to show that LLMs can automatically create zero-shot prompts, and optimize over manually written prompts. As presented in Table 1 of the paper, we demonstrated that the best prompt automatically produced by our approach achieved 80.2% accuracy on GSM8K with PaLM 2-L, which significantly outperforms zero-shot chain-of-thought prompting with “Let’s think step by step” (71.8%), and matches the few-shot chain-of-thought prompting performance (80.7%, as shown in PaLM-2 Technical Report https://arxiv.org/pdf/2305.10403.pdf, Page 15 Table 7), where the few-shot chain-of-thought prompt was manually written.

The process of “copying execution outputs into the prompt” is part of a wide variety of approaches, including ReAct, LLMs for tool use (e.g., https://arxiv.org/abs/2302.04761), LLMs for code generation (e.g., https://arxiv.org/abs/2304.05128), etc. Including this process itself does not degrade the novelty of our work. Instead, our core contribution is to show that LLMs can learn to optimize from the trajectory of solution-score pairs, which is fundamentally different from ReAct that utilizes LLMs to interact with the knowledge base or an embodied environment for decision making.

Reviewer comment (Evaluation):
How do the results compare?

Author response:
These works are complementary to our work. All 3 works used *manually written prompts* to improve LLMs’ performance, while our method focuses on *how to automatically find such prompts via optimization* and in many cases we can find even better prompts compared to the manually written ones in these existing works. 

More specifically, https://arxiv.org/pdf/2205.11916 shows LLMs can do zero-shot chain-of-thought prompting; https://arxiv.org/abs/2201.11903 shows chain-of-thought prompting improves LLMs’ reasoning performance; https://arxiv.org/abs/2305.10601 shows tree-of-thought improves the performance on several tasks that require planning and search. These papers are about manually improving the LLM performance with manually written prompts, while our approach is on prompt optimization/tuning. For example, our work showed how our method greatly improved zero-shot chain-of-thought prompting.