Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together (2024)

Abstract: Natural Language Processing (NLP) systems are increasingly taking the form of sophisticated modular pipelines, e.g., Retrieval Augmented Generation (RAG), where each module may involve a distinct Language Model (LM) and an associated prompt template. These compound systems often lack intermediate labels or gradient flow to optimize each module, making their end-to-end optimization challenging. Here we seek strategies to optimize both the module-level LM weights and the associated prompt templates of such systems to maximize a downstream task metric. We propose for the first time combining the weight and prompt optimization strategies to optimize a modular LM pipeline by alternating between the two to get the same LM to teach itself. In experiments with multi-hop QA, mathematical reasoning, and feature-based classification using mistral-7b, llama-2-7b, and llama-3-8b, these BetterTogether strategies optimizing the weights and prompts of a pipeline together outperform directly optimizing weights alone and prompts alone by up to 60% and 6%, respectively, on average across LMs and tasks. Our BetterTogether optimizer is released in DSPy at [http://dspy.ai](http://dspy.ai).

## introduction while the capabilities of language models (lms) continue to grow, recent work has shown the potential of building more powerful natural language processing (nlp) systems by composing multiple skills of lms into pipelines. examples of this include systems for retrieval-augmented generation @xcite @xcite , multi-hop reasoning @xcite , information extraction @xcite , and other sophisticated pipelines @xcite @xcite @xcite . such lm programs offer much more control for designing nlp systems, as they break down problems into modular, more manageable sub-tasks that can be assigned to lms. if we could teach these lms to accurately conduct their easier subtasks and to communicate effectively within multistage pipelines, this could greatly expand the scope of reliable nlp systems we can build. to this end, @xcite introduced the dspy framework for defining and optimizing lm programs. in it, a program is defined as a function φ that composes a set of stages, which we will refer to as language modules @xmath6, . . . , m |m | ⟩, into a pipeline. each language module m i specifies a fuzzy natural-language transformation (e.g., generating a summary of a supplied document) that needs to be learned. to do so, each module learns a particular prompt (template) π to make a call to a particular lm with weights θ. the optimization problem is then defined as maximizing the expected performance (per a downstream metric µ) of the program φ over a set of inputs by updating each module's π and θ. existing work @xcite has studied optimizing the discrete string prompt of each module and has considered simple approaches for fine-tuning each module's lm weights. in this empirical study, we investigate updating each module's lm weights and prompt template together to maximize a downstream metric on the final output of the program. doing this is challenging as φ is not generally differentiable and its modules m i generally lack labeled outputs, while exhibiting sophisticated dependencies. moreover, in realistic settings, the training set is usually very small and only a small number of lm calls are possible for training and inference. to address this challenge, we propose to alternate between fine-tuning lm weights and optimizing prompt templates and evaluate approximate optimization strategies in which we bootstrap training labels for all pipeline modules. in experiments with multi-hop qa (hotpotqa), mathematical reasoning (gsm8k), and feature-based classification (iris), we show that these tandem strategies are highly effective across three different lms, leading to 5-78% gains for hotpotqa, 2.5-10% gains for gsm8k, and 3.5-88% gains for iris against prompts only and weights only strategies, averaged across mistral-7b-instruct-v0.2, llama-2-7b-chat, and llama-3-8b-instruct. ## problem statement we are given an lm program φ, which operates like a blackbox function φ : @xmath7, in which x and y are typically in natural language (e.g., questions and their program-generated answers, respectively). for example, we may have a program φ for answering complex questions with short factoid answers. in the course of its execution, φ makes one or more calls to each of its |m | ≥ 1 language modules, @xmath8, . . . , m |m | ⟩. for example, the program may implement a multi-hop, retrieval-augmented pipeline for question answering. this common pipeline @xcite @xcite breaks down the input into subquestions that are used to iteratively find relevant passages (e.g., from a corpus like wikipedia) until the question can be faithfully answered. in general terms, each module m i : x @xmath9, in inherently fuzzy natural-language terms, an input x i domain (like a user-supplied question and a set of retrieved passages) and an output y i co-domain (like a search query to find additional relevant passages). we seek to implement each language module as some specific, well-tuned strategy for invoking an underlying language model lm. concretely, we assume that a module m i will be fully implemented by specifying (1) the string prompt π i in which the module inputs x i are plugged in to decode the module outputs y i and (2) the floating-point weights θ i assigned to the parameters of lm in the course of this module. we refer to the version of φ in which the prompts and lm weights are assigned explicitly to π and θ, respectively, as φ ⟨θ,π⟩ . given nothing but a small training set @xmath10, m 1 ), . . . , (x |x| , m |x|) )} of inputs x @xmath11, and a metric µ : y × @xmath12, our goal is to optimize φ, that is, configure its modules' prompts and lm weights to maximize the following objective. researchers tuning lm pipelines are in effect seeking to achieve this objective. it is also a very large subspace of the optimization problem in the dspy framework foot_0 for lm programs. unfortunately, this problem is intractable: the search space is large and we don't have gradients or intermediate output labels to optimize each module, so we seek approximate strategies for such optimization. ## bettertogether: alternating weight and prompt optimization steps for lm programs we now introduce the bettertogether algorithm, which alternates the weight and prompt optimization steps for lm programs. we hypothesize that, when a large lm is used to teach itself how to tackle the task defined by an lm program, finetuning lm weights and prompts are both essential to achieve the highest quality. in particular, we expect that (1) prompt optimization before finetuning can lead to more successful datapoints for fine-tuning, and, (2) prompt optimization after finetuning can make adjustments to the behavior of the lm program, leading to higher quality outputs. considering that fine-tuning is often perceived as a more powerful tool, this can be surprising, especially when both approaches are ultimately applied over the same set of training inputs x. algorithm 1 bettertogether: optimizing lm programs by alternating prompt and weight optimization steps, instantiated in algorithm 2 input: program φ ⟨θ,π⟩ = φθ ⊙ φπ, with module weights θ = [θ1, . . . , θ |φ| ] and module prompts π = [π1, . . . , π |φ| ] training set x and metric µ 1: function bettertogether(φ ⟨θ,π⟩ , x, µ) 2: accordingly, the general optimization framework for our algorithm is defined in algorithm 1. given a program φ, the algorithm begins by optimizing φ's prompts, then fine-tuning its set of lm weights with the data bootstrapped using the optimized prompts, and finally optimizing its prompts again using the fine-tuned weights. in principle, each of these steps could be treated as optional. this will define the different possible combinations of bettertogether that we will seek to evaluate in section 4. specifically, we are interested in the quality of (1) the vanilla program φ with simple user-supplied instructions as the prompts and no fine-tuning of lm, (2) optimizing the prompts only, (3) optimizing the weights only, (4) optimizing the prompts twice, i.e., using the promptoptimized φ as a starting point for a second round of prompt optimization, ( 5 ) optimizing the weights twice, (6) optimizing the prompts then the weights, (7) vice versa, and (8) optimizing the prompts, weights, then prompts. overall, we expect the final three to consistently outperform the first five. for algorithm 1 to be complete, we need to instantiate lines 1-3 with specific approaches for prompt optimization and lm fine-tuning. for this, we choose the bootstrap- * family of algorithms from @xcite , which work by executing an initial version of the program on input examples (x i , m i ) ∈ x and recording the inputs/outputs observed at each module when the final output is "correct", i.e., µ(φ(x i ), m i ) ≥ λ for some threshold λ (e.g., 1.0 for binary accuracy). this is important to note: in line with our problem formulation, our prompt and weight optimization regimes are not simply training on hand-labeled data but on self-generated program traces. instantiations for lines 1-3 of algorithm 1 are shown in algorithm 2. for prompt optimization, we use bootstrapfewshotrs (bfrs) of dspy, which self-generates potential few-shot examples of every module and applies a form of random search (rs) to select the specific generated fewshot examples that are used for prompting. overall, bfrs first divides x into a training split t and a validation split v (line 2). it then executes the provided φ ⟨θ,π⟩ on the training inputs, collecting input-output pairs for every module in φ for each x @xmath13. this is called a trace τ , and we keep only the traces assigned high scores by µ (line 4). given all of these traces, bfrs samples multiple different subsets of a few traces τ ′ (line 6), each of them containing a potential few-shot example for each module in φ, and ultimately selects the subset that, when used to construct few-shot prompts ## experimental evaluation we now seek to evaluate our hypothesis on the importance of optimizing both lm weights and prompts of lm programs. we conduct our evaluation across three datasets that span different tasks (and thus lm programs) each. in particular, we use hotpotqa @xcite for multi-hop reasoning, gsm8k @xcite for arithmetic reasoning, and iris @xcite for classification. we run our experiments using three models, mistral-7b-instruct-v0.2 @xcite , llama-2-7b-chat @xcite , llama-3-8b-instruct (metaai, 2024), keeping the model used for prompt optimization, bootstrapping training traces, and fine-tuning the same in a given experiment for all modules. we initialize all the modules of an input program φ to use the same lm weights (e.g. mistral-7b-instruct-v0.2), but distinct prompt templates specialized for their particular module-level task, such as generating a search query or answering a question. we implement all of our programs and optimizers as extensions to the dspy framework. for each dataset, we split the data into training, development, and test sets. we shuffle the training set every time before we perform a prompt or weight optimization, controlled by random seed. for prompt optimization, we sub-sample non-overlapping sets of training and validation examples from the initial training set for each task and use the bfrs prompt optimizer to optimize the module level prompts of a given φ, leaving its underlying lm weights untouched. for weight optimization, we use all the available training examples as potential candidates to generate the data for finetuning φ's lm weights and pass them to the bft weight optimizer, which: (1) runs a given φ on all the training examples, (2) keeps the traces where the final output of φ was correct and filters out the rest, (3) gets module level prompt-completion pairs for each trace, (4) creates new pairs by replacing the prompt generated by φ with a vanilla prompt, ( 5 ) combines all the module level promptcompletion pairs into one dataset, (6) fine-tunes φ's lm weights on this dataset, and finally, (7) returns an updated φ where the lm weights of all the modules are set to the fine-tuned lm. we use the low rank adaptation (lora; @xcite ) method to fine-tune our lms. the full text for our programs and the vanilla prompts are shared in appendix a. the license information for all lms and datasets used as well as our implementation details such as hyperparameters and software are reported in appendix b and appendix c, respectively. multi-hop reasoning hotpotqa (in the "fullwiki" setting) is a question answering task in which systems must find two wikipedia page abstracts out of a corpus of 5 million via search and use them to answer a factoid question. therefore it can be implemented as a program that has three lm modules: the first two for generating search queries (i.e., hops) and the last one for generating an answer. each module uses chain-of-thought (cot; @xcite to generate its outputs, producing a reasoning string before the search query or the answer. search queries are passed to a frozen colbertv2 @xcite retriever. accuracy is measured using the exact match score of the answer with the ground truth answer for the given question, after normalizing case, stripping surrounding whitespace characters, and removing punctuation. we use the following splits for hotpotqa: 1000 training examples and 500 development examples drawn from the original training set, along with 1500 test examples drawn from the original validation set, since the original test set is not public. we sub-sample 100 and 250 nonoverlapping examples for the prompt optimization training and validation sets, respectively. arithmetic reasoning gsm8k is a benchmark consisting of grade school math problems. we implement it as an lm program with a single module using cot prompting, where the lm generates a reasoning string followed by an answer. we measure accuracy by extracting the last number from the first line of the model's response and comparing it to the ground truth response. for gsm8k, we use a training set of 1000 examples and a development set of 500 examples, both sampled from the original training set. we use all the 1319 test examples available for our test set. we use 100 and 250 nonoverlapping examples for training and validation sets used during prompt optimization. classification iris is a classic classification task, where the goal is to classify species of iris flowers. we use a single-module cot dspy program for iris, with the goal of assessing whether it being a feature-based classification task gives a large advantage to methods based entirely on gradient descent (fine-tuning). this tests the extrapolation of our hypothesis to a different setting from the other two tasks. we measure accuracy using the exact match score of the answer and the correct answer given a question after normalizing both, as is the case for hotpotqa. the iris dataset has a total of 150 examples, from which we create training, development, and test splits of equal size. we sub-sample 15 and 35 non-overlapping examples to be used as the training and validation sets for prompt optimization, respectively. ## results & discussion held-out test set performance of the strategies described in section 3 is shared in in 7 out of the 9 dataset and lm pairs, we observe that the best-performing strategies are the strategies that utilize prompt (π) and weight (θ) optimization steps together, although there is no clear winner among the benchmarked bettertogether strategies that optimize both. overall, optimizing prompts is essential on all the tasks, but optimizing prompts and weights together leads to strong gains over the best setting that only optimizes one of the two. in summary, we have proposed to alternate between optimizing prompts and fine-tuning lm weights and explored a few strategies for doing so. in experiments with multi-hop qa (hotpotqa), mathematical reasoning (gsm8k), and feature-based classification (iris), we show that our strategies are highly effective for getting an lm to teach itself to perform an lm program via bootstrapping, leading to 5-78% gains for hotpotqa, 2.5-10% gains for gsm8k, and 3.5-88% gains for iris. ## limitations while this paper presents strong evidence from nine cases, spanning three tasks (and their corresponding lm programs) and three lms, it is possible that other tasks, programs, or lms will change the pattern in unforeseen ways. in particular, we have only experimented with weight optimization in the form of lora fine-tuning of pre-trained models. it is in principle possible that some other fine-tuning strategy would be so powerful and cost-effective as to remove the need for prompt optimization. in addition, though we expect our findings to inform many researchers and practitioners interested in optimizing lm programs, and encourage them to explore optimizing prompts and fine-tuning lm weights together, we do not yet understand why both are important. the role of prompt optimization and the role of fine-tuning in multi-stage lm programs are both new, and the relative lack of understanding of these roles in the emerging literature could pose risks in unanticipated interactions between these components, compared with standard gradient descent for neural networks, which has been studied for decades. table a.1: vanilla (zero-shot) prompts for the hotpotqa program given in python snippet a.1 with three lm modules, each with corresponding prompts and lm weights. (1) the generate_query[0] module prompts the corresponding lm to generate a search query given a question along with an empty context; (2) the generate_query[1] module prompts the corresponding model to generate a search query given the context paragraphs retrieved using the search query outputted by the prior module; module label prompt generate_query[0] given the fields `context`, `question`, produce the fields `search_query`. context: @xmath0{question} reasoning: let's think step by step in order to @xmath1{search_query} ---context: n/a question: what animal subject to the foster's rule is the smallest north american deer? reasoning: let's think step by step in order to find the smallest north american deer subject to the foster's rule. context: @xmath2{question} reasoning: let's think step by step in order to @xmath3{search_query} ---context: [1] «key deer | the key deer ("odocoileus virginianus clavium") is an endangered deer that lives only in the florida keys. it is a subspecies of the white-tailed deer ("o. virginianus"). it is the smallest north american deer.» @xcite «north american least shrew | the north american least shrew ("cryptotis parva") is one of the smallest mammals, growing to be only up to 3 inches long. the north american least shrew has a long pointed snout and a tail never more than twice the length of its hind foot. it has a dense fur coat that is either grayish-brown or reddish-brown with a white belly. its fur becomes lighter in the summer and darker in the winter. although similar in appearance to several species of rodents, all shrews are members of the order soricomorpha and should not be mistaken for a member of the rodentia order. the north american least shrew's eyes are small and its ears are completely concealed within its short fur, giving it very poor eyesight and hearing.» @xcite «wynnea americana | wynnea americana, commonly known as moose antlers or rabbit ears, is a species of fungus in the family sarcoscyphaceae. this uncommon inedible species is recognizable by its spoon-shaped or rabbit-ear shaped fruit bodies that may reach up to 13 cm tall. it has dark brown and warty outer surfaces, while the fertile spore-bearing inner surface is orange to pinkish to reddish brown. the fruit bodies grow clustered together from large underground masses of compacted mycelia known as sclerotia. in eastern north america, where it is typically found growing in the soil underneath hardwood trees, it is found from new york to michigan south to mexico. the species has also been collected from costa rica, india, and japan.» question: what animal subject to the foster's rule is the smallest north american deer? reasoning: let's think step by step in order to produce the search query. we need to find the smallest north american deer, which is mentioned in the context as the key deer. therefore, the search query should include the terms "key deer" and "smallest north american deer". search query: key deer smallest north american deer generate_answer given the fields `context`, `question`, produce the fields `answer`. context: @xmath4{question} reasoning: let's think step by step in order to @xmath5{answer} ---context: [1] «key deer | the key deer ("odocoileus virginianus clavium") is an endangered deer that lives only in the florida keys. it is a subspecies of the white-tailed deer ("o. virginianus"). it is the smallest north american deer.» @xcite «north american least shrew | the north american least shrew ("cryptotis parva") is one of the smallest mammals, growing to be only up to 3 inches long. the north american least shrew has a long pointed snout and a tail never more than twice the length of its hind foot. it has a dense fur coat that is either grayish-brown or reddish-brown with a white belly. its fur becomes lighter in the summer and darker in the winter. although similar in appearance to several species of rodents, all shrews are members of the order soricomorpha and should not be mistaken for a member of the rodentia order. the north american least shrew's eyes are small and its ears are completely concealed within its short fur, giving it very poor eyesight and hearing.» @xcite «wynnea americana | wynnea americana, commonly known as moose antlers or rabbit ears, is a species of fungus in the family sarcoscyphaceae. this uncommon inedible species is recognizable by its spoon-shaped or rabbit-ear shaped fruit bodies that may reach up to 13 cm tall. it has dark brown and warty outer surfaces, while the fertile spore-bearing inner surface is orange to pinkish to reddish brown. the fruit bodies grow clustered together from large underground masses of compacted mycelia known as sclerotia. in eastern north america, where it is typically found growing in the soil underneath hardwood trees, it is found from new york to michigan south to mexico. the species has also been collected from costa rica, india, and japan.» [4] «pudú | the pudús (mapudungun "püdü" or "püdu", spanish: pudú , ] ) are two species of south american deer from the genus pudu, and are the world's smallest deer. the name is a loanword from mapudungun, the language of the indigenous mapuche people of southern chile and south-western argentina. the two species of pudús are the northern pudú ("pudu mephistophiles") from venezuela, colombia, ecuador, and peru, and the southern pudú ("pudu puda"; sometimes incorrectly modified to "pudu pudu") from southern chile and south-western argentina. pudús range in size from 32 to tall, and up to 85 cm long. as of 2009, the southern pudu is classified as near threatened, while the northern pudu is classified as vulnerable in the iucn red list.» question: what animal subject to the foster's rule is the smallest north american deer? reasoning: let's think step by step in order to find the answer. we know that the smallest north american deer is mentioned in the context. we also know that the foster's rule refers to a group of animals. in this case, the foster's rule refers to endangered deer species that live only in the florida keys. therefore, the key deer is the answer.