Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs (2024)

Abstract: Language Model Programs, i.e. sophisticated pipelines of modular language model (LM) calls, are increasingly advancing NLP tasks, but they require crafting prompts that are jointly effective for all modules. We study prompt optimization for LM programs, i.e. how to update these prompts to maximize a downstream metric without access to module-level labels or gradients. To make this tractable, we factorize our problem into optimizing the free-form instructions and few-shot demonstrations of every module and introduce several strategies to craft task-grounded instructions and navigate credit assignment across modules. Our strategies include (i) program- and data-aware techniques for proposing effective instructions, (ii) a stochastic mini-batch evaluation function for learning a surrogate model of our objective, and (iii) a meta-optimization procedure in which we refine how LMs construct proposals over time. Using these insights we develop MIPRO, a novel algorithm for optimizing LM programs. MIPRO outperforms baseline optimizers on five of seven diverse multi-stage LM programs using a best-in-class open-source model (Llama-3-8B), by as high as 13% accuracy. We have released our new optimizers and benchmark in DSPy at [http://dspy.ai](http://dspy.ai).

## introduction solving complex tasks with language models (lms) often requires applying sophisticated prompting techniques @xcite and chaining them together into multi-stage pipelines @xcite @xcite @xcite . such language model (lm) programs continue to advance nlp tasks @xcite @xcite through systematic composition (khattab et al., 2021; creswell * equal contribution. and shanahan, 2022; @xcite and tool use @xcite . however, lm programs today are commonly designed via "prompt engineering": crafting prompts via manual trial and error to coerce a specific lm to operate each step in a specific pipeline. recent work such as ape @xcite , opro @xcite , and evoprompt @xcite presents prompt optimizers, i.e., algorithms that search over strings to identify high-performing prompts. unfortunately, the majority of this work does not directly apply to multi-stage lm programs in which we lack gold labels or evaluation metrics for the individual lm calls. @xcite study how to express arbitrary lm pipelines such that the instructions, demonstrations (input/output examples), and lm weights of each lm call are treated as parameters that can be optimized toward any metric. while the authors present optimizers that can create demonstrations of multi-stage pipelines and use them to optimize prompts, weights, or even both together @xcite , their proposed optimizers cannot tune instructions for multi-prompt pipelines. we seek to efficiently optimize prompts in arbitrary lm programs, especially those with multiple stages (figure 1 ) and explore approaches that hold under weak assumptions, consistent with the abstractions from the dspy programming model @xcite . in particular, we assume no access to lm weights, log-probabilites, or handwritten metrics or labels for intermediate stages in a chain of lm calls. we require only the lm program itself, a metric to optimize, and a training set of inputs (and, depending on the metric, final outputs). we formally define the problem of prompt optimization for lm programs and outline the design space by identifying two key challenges. first, the proposal challenge: the space of possible prompts is intractably large, and this is exacerbated as the number of modules increase. proposing a few highquality instructions is thus essential. second, the credit assignment challenge: our problem requires jointly optimizing over many distinct variables that parameterize the prompts of all modules. to allocate search effort, we must infer the impact of our configurations for each variable effectively. we define several strategies to tackle each of these challenges and systematically explore the tradeoffs they present. we find that optimizing bootstrapped few-shot examples is often essential for realizing the greatest performance gains, but that optimizing instructions becomes more essential for tasks with conditional rules. we also find that optimizing both instructions and few-shot examples together generally leads to the best results. we make three contributions. first, we present a formalization of the problem of optimizing language model programs ( §2) and propose an algorithm design space with three strategies to address the challenge of prompt proposal and three strategies to resolve the issue of credit assignment ( §3). second, we release a benchmark suite for lm program optimizers spanning seven tasks ( §5). third, we construct and evaluate a rich subset of possible algorithms for prompt optimization ( §4). highlighted amongst these algorithms is mipro (multiprompt instruction proposal optimizer) which outperforms baseline optimizers on five of seven tasks in our benchmark, by as much as 13% accuracy improvement. using our algorithms, we derive five key lessons for practitioners looking to optimize lm programs ( §6). ## problem statement algorithm 1 optimize φ with optimizer m 1: input: optimizer m , initial program φ, metric µ 2: input: max iterations i, training data d 3: input: minibatch size b, proposer hyperparameters θ 4: output: optimized version of φ 5: 6: m .initialize(d, θ) ▷ initialize optimizer using the data 7: for k ← 1 to i do 8: m .update(@xmath0, σ) ▷ update optimizer based on the observed validation score 12: end for 13: (@xmath1.extractoptimizedsets() 14: return φ @xmath2, each using some lm. each module i is defined by a prompt template p i that contains a set of variables (open slots) v. for example, a prompt template for few-shot qa might have variables for instructions, demonstrations, and the target question. let v be the set of all variables used by prompt templates for φ, and let @xmath3. we use φ @xmath4. our high-level goal is to find a total assignment that optimizes φ's performance with respect to metric µ on a trainset d that has inputs x and optional metadata x ′ (such as labels): this is the problem faced by people designing lm programs. it is intractable, as (i) each string @xmath5, (ii) the metric µ provides supervision only at the level of the entire task, so every variable in v is latent, and (iii) we assume no access to the gradients or embeddings of the lms involved, which rules out many rl and prompttuning algorithms @xcite @xcite . in addition, (iv) system designers generally have small datasets d and (v) small budgets of lm calls for evaluating φ. in many cases, we want to optimize just a subset of the variables used by φ. in the present work, for example, we assume each prompt p has a variable i over free-form instructions and a set of k variables {d i1 , . . . , d ik } over demonstrations. in these set-tings, we assume that all the other variables for φ are set to constant values. to find approximate solutions to ( 2 ), we work within the general optimization framework defined by algorithm 1. this framework generalizes prior approaches such as opro @xcite and ape @xcite to optimizing lm programs. the main parameters to this method are the optimizer m and the unoptimized program φ. we assume that each optimizer has methods initialize, propose, update, and extractoptimizedsets and that it has some internal state which informs proposals, and updates on calls to update. ## designing lm program optimizers algorithm 1 defines a general optimization framework for lm programs. we seek efficient instantiations of this algorithm in which we minimize the number of times the program φ is invoked (line 10) and, as a result, the number of times we must sample proposals (line 8). to this end, we are especially interested in building lm program optimizers with strategies that handle the proposal and credit assignment challenges discussed in section 1. in this section, we present several novel or improved strategies for this. section 4 then defines a few effective compositions of these strategies that allow us to empirically study their properties in practice. we showcase how these components come together to make lm program optimizers in figure 4 . ## optimizers we now motivate specific instantiations of optimizer algorithms composed of the algorithmic strategies outlined in section 3. we describe the algorithms that are the focus of our experimental investigation, and leave full descriptions for the remaining optimizers to appendix f. ## results & discussion lesson 1: optimizing bootstrapped demonstrations as few-shot examples is key to achieving the best performance. for the majority of tasks, we find that optimizing boostrapped demonstrations alone yields significantly better performance than optimizing instructions alone. we confirm this with a wilcoxon signed-rank statistical test, which shows that even simple bootstrap random search beats the best instruction-only optimizer for a given task in all but one case. the exception to this is hotpotqa conditional, which supports our hypothesis and findings discussed in lesson 3. we finally note that creating the right set of bootstrapped demonstrations is important. our optimization runs indicate that there is high variation in the performance resulting from different few-shot sets (see appendix g). we thus infer that strong bootstrapped examples provide information pertaining to successful reasoning behavior more than just, say, teaching task format. lesson 2: optimizing both instructions and few-shot examples with mipro generally yields the best overall performance. we support this with a statistical test comparing mipro with the second highest averaging optimizer for each task. the exceptions to this are hotpotqa, heart disease, and iris without a typo. we hypothesize that instructions are less valuable for tasks like hot-potqa, whose final module is a relatively straightforward q&a task that is likely in-distribution for many models. for heart disease, we hypothesize that this is due to initializing our optimizers with a simple seed instruction that does not convey any classification criteria, which current instruction optimizers have a limited ability to infer. lesson 3: instruction optimization is most important for tasks with conditional rules that are (i) not immediately obvious to the lm and (ii) not expressible via a limited number of fewshot examples. this hypothesis is supported by our iris-typo experiment-and primarily by our hotpotqa conditional experiments, where we optimize over a seed instruction stating the rules of the task. for this task, we find that even 0-shot instruction optimization outperforms demonstration-only optimization. in these cases, especially if the task is complex, it's important that we optimize over a seed prompt, as our optimizers are not yet able to infer all task rules. (this lesson is also reflected in the heart disease results, as discussed above.) in the iris-typo setting, our instruction optimizer even helps correct mistakes in the seed prompt. lesson 4: grounding is helpful for instruction proposal overall, but the best proposal strategy varies by task. in our module-level opro grounding ablations, we find that grounding is essential for performance improvements for hotpotqa and hover, but seems to hurt performance for scone. this motivates approaches like mipro++, which are able to learn custom proposal strategies for a given task. indeed, we see that 0-shot mipro++ is able to learn a proposal strategy that recovers this performance for scone. one additional benefit of 0-shot mipro++ is that the learned importance scores from the bayesian model used to optimize proposal hyperparameters can provide us insight into the utility of each proposal component. studying these importance scores (found in appendix d) reveals, across tasks, the highest importance scores go to the choice of bootstrapped demonstrations in the meta-prompt and the tip. we observe that the importance of many parameters varies between tasks: for example, the dataset summary has a high learned importance score for scone, whereas it is one of the least lesson 5: there is more to learn about lm program optimizers. when comparing the performance of module-level opro, 0-shot mipro, and 0-shot mipro++, we find that results are mixed. bayesian bootstrap outperforms bootstrap random search for scone, but this finding is not statistically significant for hotpotqa and hover. 0-shot mipro++ outperforms 0-shot mipro for scone and hotpotqa, but is about equivalent in the case of hover. future work may find more differentiated results when studying these optimizers at different optimization budgets: for example, it's possible that 0-shot mipro would perform best in very low budget settings, given that it's use of minibatching allows it to explore many more parameter options with the same budget. conversely, 0-shot mipro++ may shine in scenarios where budget is not an issue, and spending many trials to learn optimal proposal dynamics could lead to differentiated results. we save this exploration for future work. ## related work recent work has explored optimizing string prompts, including gradient-guided search @xcite , reranking brute force search @xcite , evolutionary algorithms @xcite , prompting other lms @xcite @xcite , and reinforcement learning (rl) @xcite @xcite . prior work on rl for prompts focuses on word level edits of only a few words @xcite , phrase level edits @xcite , or text-to-image generation @xcite . @xcite present dspy, a programming model for expressing lm programs and optimizing their prompts and weights. unlike our work, the authors only explore optimizers based on bootstrapping strong demonstrations. @xcite explore joint prompt optimization for stacked llm calls, modeling this as variational inference and exploring this for two simple layers. their approach inherently relies on having access to log probabilities for explicitly passed tokens to lms, an increasingly restrictive assumption in practice (e.g. with commodified lm apis that allow only text-in-text-out processing). in contrast, our optimizers work on an arbitrary number of modules for any lm program out-of-the-box. ## conclusion we formalize the problem of optimizing prompts in lm programs ( §2). we identify two key challenges of lm program prompt optimization: (1) proposal of a small set of high-quality prompts and (2) credit assignment during optimization. we address these challenges using three proposal generation and three credit assignment strategies ( §3) and explore a representative subset of optimizer algorithms ( §4) using a new benchmark of diverse tasks ( §5.1). our findings show that optimizing few-shot demonstrations is very powerful, but instruction optimization can be essential for complex task specifications with multiple conditional rules. finally, we find that jointly optimizing both instructions and demonstrations using the mipro optimizer is the most effective approach in five out of seven settings.