Paper: DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines (ICLR-2024)

Reviewer comment (Reproducibility):
Lack of details: DSPy stands out from other frameworks, like LangChain, due to its capacity for automatic prompt generation and optimization. However, the introduction of the DSPy Compiler lacks sufficient specificity. It would be beneficial to provide more comprehensive details on how DSPy generates candidate values for module parameters (instructions, field description, and example input/output) based on signatures.

Author response:
Thank you for the suggestion. We now include a new appendix I (pages 28, 29, 30) which include snippets of our optimized prompts, when compiling with Llama2-13b-chat. This should help the reader concretely visualize the parts of a typical prompt constructed from a signature. In particular, it contains the instructions (which, if omitted, are inferred from the fields), the field organization, and the demonstrations.

In short, the module defines a parameter space. A teleprompter can then optimize these different pieces. For instance, we have an ongoing research project that optimizes the instructions of DSPy modules automatically. In this paper, we focus on optimization with random search over demonstrations, where we automatically created traces of each module, and then select different traces as few-shot examples (or finetuning training data), even if there are no hand-written labels for these modules.

Reviewer comment (Reproducibility):
Can we easily implement common and advanced prompting techniques with signatures? I noticed that the signatures used in the Case Study are quite simple. When it comes to dealing with complex user intentions, do you think additional hand-written comments are necessary alongside signatures? For instance, in the ChainOfThought module, I found that the prompt "Let's think step by step" still requires handwriting within the module. How many prompts would need to be handwritten to achieve functional equivalence with a complex prompt (such as Appendix D prompt 6) using signatures?

Author response:
A prompting technique is best abstracted as a Module in DSPy. It is indeed possible that a specific approach to prompting requires adding new modules to DSPy, which should work for any signature (i.e., any input/output patterns).

Our vision is that any minor hard-coded parts of the modules (e.g., “Let’s think step by step.”) should eventually be treated as variables to optimize. Nonetheless, like in traditional DNNs, there are always hyperparameters that may be selected by hand, as long as the overall model at the end is optimized automatically.

Since the submission of this work, we have built (or helped others build) a number of new sophisticated systems with DSPy. In some of them, we found that users like to include additional comments (or instructions) in the extended DSPy format that we present in Appendix B “Advanced Signatures”. To make this more robust, we have been exploring techniques to automatically optimize these prompts and to allow for automatic backtracking logic to correct errors. Considering the scope, length, and complexity of this paper, we leave exploring these additional components to future publications related to DSPy.

Reviewer comment (Reproducibility):
How can users debug their DSPy programs effectively? In Section 4 of the paper, it is mentioned that "A well-decomposed program can typically find at least a few training examples..." I suppose that the success of optimization depends on a well-structured decomposition by the user, which implies the need for frequent modification of DSPy programs. Consequently, it raises the question of how users can determine which modules are not functioning correctly. Is it necessary for users to repeatedly rewrite the entire DSPy program?

Author response:
One of the key reasons we built DSPy as a “define-by-run” abstraction (like PyTorch, or the “eager mode” of TensorFlow 2.0) is to facilitate debugging.

In particular, users of DSPy can simply insert checkpoints or use print statements between different steps of their DSPy programs. For example, in the multi-hop program, we can print or log every search query generated (or even manually change these queries) in order to understand which module is responsible for different mistakes.

Because it is possible to debug one module at a time, we do not anticipate any need for users to repeatedly rewrite their programs. Instead, they can focus on isolating individual types of errors and then improving the responsible module. Of course, DSPy is not a panacea and sometimes development or debugging may require more end-to-end tracing of the full program. To this end, we recently started adding support for tracing of the full program to visualize all the prompts during execution.

Reviewer comment (Evaluation):
How much does the optimization cost in terms of computing resources or API usage? (This question may exceed the paper's scope)

Author response:
Thank you for emphasizing this element. To address this, we have added a new appendix A on Latency & Throughput of using DSPy. To illustrate that the cost of using DSPy is manageable, we consider a sophisticated program from our evaluation, i.e. the multi-hop search and QA program. We report on compiling it with OpenAI GPT-3.5 for $3.00 in just six minutes. Inference with this program takes approximately 3 seconds per example (single-threaded) and can easily exceed 150 questions per minute (with ten DSPy threads).

Future work should explore efficiency in a much broader evaluation to characterize this emerging space of LM programs. Just like the cost of training a new model, the cost of compiling a DSPy can vary greatly depending on the complexity of the program and the optimizer. However, we hope this small-scale evaluation sheds some light on the typical cost of compiling a program like this.

Reviewer comment (Experiments):
In section 5, Hypothesis 2, the paper mentions the possibility of DSPy outperforming expert-written prompts. However, is there an evaluation comparing DSPy with handwritten prompts to support this claim?

Author response:
This is indeed an important aspect of DSPy. In our evaluations, the CoT program for GSM8K includes comparison between the “bootstrap” compilation approach and the “fewshot +human_CoT” setting (Table 1). The former generates its own optimized few-shot examples (including, in this program, the reasoning chains). The latter samples expert-written step-by-step reasoning chains, which are the typical way other papers build prompts for GSM8K (e.g., the original CoT paper by Wei et al). Another instance of this is the ReAct evaluations in Table 2, which compare handwritten prompting against our bootstrapping compilation approaches. Across these evaluations, we found that the compiled systems match or exceed the quality achieved by hand-written prompts and reasoning chains. These results can also be compared with systems from the literature (e.g., the original ReAct scores on HotPotQA), which generally use hand-written prompts (e.g., see Appendix E).

Reviewer comment (Reproducibility):
In Section 5 of the paper, it states, "... ‘how they compare on GSM8K with program P when compiled with strategy S’, which is a well-defined and reproducible run.” How can the reproducibility of these results be ensured?

Author response:
To reproduce a specific program, DSPy supports saving a dump of the compiled program. This is akin to saving a trained model, to draw an analogy with deep neural networks. As long as the LM can produce reproducible outputs, loading the saved program can be used to replicate old runs, if the LLMs are set for greedy decoding (i.e., temperature=0.0). DSPy also supports caching (and sharing the cache for) all invocations of the LM(s), which is another way to ensure reproducibility as needed.

More generally, it's also possible to re-run each setup several times (with higher temperature) and average the runs. Either way, like DNNs, DSPy offers a more self-contained setup than hand-drafting prompts (or hand-tuning classifier weights), which aids reproducibility of the key ideas in the the more general sense that the optimization process itself is automated and hence can be reapplied to new settings. In contrast, hand-tuning prompts is more of an art than a science and is hard to recreate.

Reviewer comment (Evaluation):
In Case Study 1 using the GSM8K dataset, GPT-3.5 with 5 few shots achieves a score of 57%, while vanilla+GPT-3.5+fewshot only scores 33.1%. What factors contribute to the decline in performance, considering the similarity between these two approaches?

Author response:
Thank you for the remark. Let us clarify how the different approaches for few-shot prompting work on GSM8K.

In Table 1, we report the following results with `gpt-3.5-turbo-instruct` (released in September 2023).

1. Using the `fewshot` (LabeledFewShot) teleprompter in DSPy, on the `vanilla` program.
2. Using the `fewshot` teleprompter on the `CoT` program.
3. Using the `fewshot` teleprompter on the `CoT` program, with the extended “+human CoT” training set.

In the first two, we prompt the LM with few-shot examples that do *not* contain reasoning chains. That is, the demonstrations in #1 and #2 only include the final numerical answer. In the first one, we also do not request the model to engage in a step-by-step reasoning chain, unlike #2 and #3. In the third one, the few-shot examples include human-written reasoning chains.

These three approaches score 33%, 65%, and 78% on our development set. We test the last one on the official test set, and it scores 72%. In this case, the most comparable run to the original 5-shot GPT-3.5 evaluation by OpenAI is the third option. As you can see, our implementation here scores more highly (72% > 57%), which may be a result of using a newer release of GPT-3.5 (September vs March 2023), using extra demonstrations (5-shot vs 8-shot), and variance across samples (we average 5 runs).

Reviewer comment (Novelty):
My impression is that the value-add of DSPy for any of the listed strategies is somewhat small. E.g. implementing any of CoT, Reflection, or ReAct without DSPy does not require much code. The same is true, I think, for the optimization approaches / compilers. I expect this value-add grows when working with many such strategies at once, and additionally that DSPy provides organizational value both as the programs and compilers grow in complexity, and as the set of people using them grows. Being able to compose different strategies and compiler techniques is also one of the key beneficial properties of the system.

Author response:
Thank you for the nuanced discussion. We agree that much of the power of DSPy comes from enabling rich compositions of modules and compilation strategies. However, we wish to emphasize that even simple programs (e.g., those that rely on a single module, like CoT or ReAct) stand to benefit from DSPy. To illustrate, it’s indeed true that expressing CoT (or, say, ReAct) does not require much code without DSPy. However, expressing them *well* for a particular task generally requires writing a good prompt, often with high-quality few-shot examples for reasoning or tool use. Adapting this prompt across LMs may also pose a challenge. While these challenges for a single, simple module are not very large (and indeed more research is needed to make general claims about them), we believe the DSPy paradigm offers a consistent way to optimize both simple and complex programs alike.

Reviewer comment (Evaluation):
The evaluations performed do not provide a measure of compute usage or time (both for compilation as well as inference of the compiled programs), which makes comparisons across programs and compilers less meaningful.

Author response:
Thank you for emphasizing this element. To address this, we have added a new appendix A on Latency & Throughput of using DSPy. To illustrate that the cost of using DSPy is manageable, we consider a sophisticated program from our evaluation, i.e. the multi-hop search and QA program. We report on compiling it with OpenAI GPT-3.5 for $3.00 in just six minutes. Inference with this program takes approximately 3 seconds per example (single-threaded) and can easily exceed 150 questions per minute (with ten DSPy threads).

Future work should explore efficiency in a much broader evaluation to characterize this emerging space of LM programs. Just like the cost of training a new model, the cost of compiling a DSPy can vary greatly depending on the complexity of the program and the optimizer. However, we hope this small-scale evaluation sheds some light on the typical cost of compiling a program like this.

Reviewer comment (Experiments):
There are no examples of compiled programs provided in the paper or appendices. I think an analysis of compiled programs would benefit the paper meaningfully. In particular, some unanswered questions about the compiled programs include: how do they differ from the types of prompt programs that people write by hand or using other frameworks? how do programs compiled for small LMs differ from those compiled for large LMs? are there any obvious patterns in the compiled programs? how about obvious shortcomings or irregularities, where additional hand-optimization would be easy? any evidence that the optimization techniques overfit to the validation set?

Author response:
We now include a new appendix I (pages 28, 29, 30) which include snippets of our optimized prompts, when compiling with Llama2-13b-chat. We believe an extensive analysis of these prompts, especially when varying LMs and varying optimization strategies, is a highly exciting future direction.

Nonetheless, it may be useful to consider Figure 11 (page 30) which reports on the Llama2-13b prompt for the second hop of query generation for HotPotQA. We see that it teaches the model two different strategies for generating search queries. The first bootstrapped few-shot example is a well-formed question "When was the first of the vampire-themed fantasy romance novels published?". The second is a keyword-based search query, "Jeremy Paxman birth year". It is possible to hypothesize this duality contributes to the selection of this prompt during optimization. Future work may try to characterize these patterns on a principled basis.


> 'how about obvious shortcomings or irregularities, where additional hand-optimization would be easy?'

One of the limitations of the BoostrapFewshot set of teleprompters we discuss in this work is that they focus on generating and selecting demonstrations of the input/output behavior of each module. To expand this toward more open-ended optimization, we have started multiple research directions that explore optimizing the instructions and execution logic (e.g., backtracking on failure) for the modules.

Reviewer comment (Writing):
In the abstract you say DSPy programs are "compiled to" a language model. I think this wording is a bit wrong or misleading, and that it would be better to say the program is "optimized for" or (if you insist on the language of compilation) "compiled for".

Author response:
Thank you for this extensive list of notes and typos. We believe we have now addressed all of these in the PDF. (We will update the submission abstract outside the PDF too, if the work is accepted. The system does not permit this change currently.)

Reviewer comment (Theory):
Teleprompters are described as general-purpose optimization strategies, though if I understand correctly they are limited to gradient-free optimization techniques. This detail warrants mentioning.

Author response:
Thank you for this extensive list of notes and typos. We believe we have now addressed all of these in the PDF. (We will update the submission abstract outside the PDF too, if the work is accepted. The system does not permit this change currently.)

Reviewer comment (Novelty):
The paper claims DSPy is the first programming model that translates prompting techniques into parameterized declarative modules that can be optimized. Several relevant prior works exist, such as Dohan 2022 Language Model Cascades, Decomposed Prompting, ANPL, and Large Language Models as Optimizers. Among these, Dohan 2022 is closest to the claim, suggesting that the novelty claim may need qualification.

Author response:
We agree to soften our statement in the introduction, in light of this note. In short, Dohan et al. (whose work we cite as early as our first paragraph) present a rich self-described “position paper” on the conceptual analogy between what they call “LM cascades” and probabilistic graphical models. While we find that work highly valuable, we are unable to immediately extrapolate from this conceptual analogy to a programming model in which prompting techniques can be optimized directly, as we do with DSPy. Nonetheless, to avoid any unintended implications, we have softened our statement in the introduction (marked in blue in page 2) in line with your comment.

Reviewer comment (Reproducibility):
There is no discussion on how to handle safety during SQL generation, given that SQL queries can modify or delete tables, nor how to approach using tools with side effects during compilation.

Author response:
These are both good examples where using DSPy does not replace the value of good, thoughtful experimental design. In particular, care must be taken if the program has access to tools that can modify (or destroy) records or execute code, like running it in a sandbox. We have a small Python execution sandbox in DSPy to aid with this, but it is not an alternative for thoughtful design.

Similarly, as with standard ML techniques, overfitting is an important consideration, especially when the training and validation sets are small. In our experience, we found limited signs of detrimental overfitting (i.e., selecting low-quality systems) but found evidence of sets of different difficulty such that the absolute scores of the systems are different across validation or evaluation sets.

For good experimental design, the teleprompters allow us to distinguish between the training, validation, development, and test sets. For instance, when compiling the multi-hop program on HotPotQA, we use the first 50 examples of the training set for training, and examples 50 through 200 for validation. We compare different approaches on a separate development set (300 examples). And we finally report the quality on the test set (1000 separate examples). While the teleprompters enable such good experimental design, this cannot be enforced. It is always possible for misuse of the optimization process to overfit.

Reviewer comment (Writing):
In the RAG example at the end of Section 3.2, it is unclear if it is complete without a definition of Retrieve provided.

Author response:
At the start of a script that uses DSPy, the user declares the “default” language model and retrieval model. These choices can also be changed for specific code blocks. In our experiments, we set the default retriever to be a ColBERTv2 (Santhanam et al., 2022) index over a Wikipedia corpus. However, this is very easy to extend. Our users have introduced many integrations for retrieval, from Pyserini to Pinecone to Weaviate, among other popular systems.

Reviewer comment (Reproducibility):
It is unclear how a DSPy user can introduce a new Retrieve implementation.

Author response:
At the start of a script that uses DSPy, the user declares the “default” language model and retrieval model. These choices can also be changed for specific code blocks. In our experiments, we set the default retriever to be a ColBERTv2 (Santhanam et al., 2022) index over a Wikipedia corpus. However, this is very easy to extend. Our users have introduced many integrations for retrieval, from Pyserini to Pinecone to Weaviate, among other popular systems.

Reviewer comment (Evaluation):
Regarding how compiled programs differ when targeting smaller LMs, a qualitative investigation with examples would be welcome.

Author response:
We now include a new appendix I (pages 28, 29, 30) which include snippets of our optimized prompts, when compiling with Llama2-13b-chat. We believe an extensive analysis of these prompts, especially when varying LMs and varying optimization strategies, is a highly exciting future direction.

Nonetheless, it may be useful to consider Figure 11 (page 30) which reports on the Llama2-13b prompt for the second hop of query generation for HotPotQA. We see that it teaches the model two different strategies for generating search queries. The first bootstrapped few-shot example is a well-formed question "When was the first of the vampire-themed fantasy romance novels published?". The second is a keyword-based search query, "Jeremy Paxman birth year". It is possible to hypothesize this duality contributes to the selection of this prompt during optimization. Future work may try to characterize these patterns on a principled basis.

Reviewer comment (Evaluation):
In Table 1, reporting some measure of compute usage or time both for compilation and inference would be valuable, and would make the various strategies (both programs and compilation approaches) more readily comparable.

Author response:
We agree that readers will benefit from a comparison of the costs associated with different programs. In principle, measuring the compilation costs requires us to re-compile our programs (as we did for Appendix A, for one program) while disabling our caches. Considering that this is a significant experimental endeavor, if our work is accepted, we will include compiling and inference costs in the camera-ready version. This can potentially serve as a new column in the results tables or an expansion of Appendix A.

We hope that the addition of Appendix A conveys the observation that the cost of compiling (which is a one-time, offline step) is fairly low. (As a ballpark, compiling is generally faster than typical finetuning jobs for instance.) Because of this, we typically recommend that DSPy users focus primarily on the quality and cost of the resulting programs, since compiling is an offline operation with manageable costs.

Nonetheless, to provide a preliminary comparison of inference costs, we took four rows of increased complexity for HotPotQA (Table 2) and tested them with gpt-3.5-1106, while disabling the cache (which would otherwise skip any repeated computations, e.g. retrieval queries or LM calls).

We ran 100 questions from HotPotQA with a single thread, and we report the average latency below. Each numbered bullet below represents a program (with compiler strategy in parentheses).

**1. Vanilla (fewshot)**

Average Latency: 0.3 seconds per question

Average Cost: $0.0005 per question (1x)

**2. CoT_RAG (fewshot)**

Average Latency: 1.1 seconds per question

Average Cost: $0.0013 per question (2.6x)

**3. Multihop (fewshot)**

Average Latency: 2.6 seconds per question

Average Cost: $0.0018 per question (3.6x)

**4. Multihop (bootstrap)**

Average Latency: 2.6 seconds per question

Average Cost: $0.0041 per question (8.2x)

As this shows, these programs are within an order of magnitude of the cost and latency of the simplest one, even though we see major quality improvements from vanilla to multihop. We hope this provides deeper insight into our evaluations, and will expand on this analysis if our work is accepted.

Reviewer comment (Experiments):
It is unclear if the compilation approaches account for overfitting the validation set.

Author response:
These are both good examples where using DSPy does not replace the value of good, thoughtful experimental design. In particular, care must be taken if the program has access to tools that can modify (or destroy) records or execute code, like running it in a sandbox. We have a small Python execution sandbox in DSPy to aid with this, but it is not an alternative for thoughtful design.

Similarly, as with standard ML techniques, overfitting is an important consideration, especially when the training and validation sets are small. In our experience, we found limited signs of detrimental overfitting (i.e., selecting low-quality systems) but found evidence of sets of different difficulty such that the absolute scores of the systems are different across validation or evaluation sets.

For good experimental design, the teleprompters allow us to distinguish between the training, validation, development, and test sets. For instance, when compiling the multi-hop program on HotPotQA, we use the first 50 examples of the training set for training, and examples 50 through 200 for validation. We compare different approaches on a separate development set (300 examples). And we finally report the quality on the test set (1000 separate examples). While the teleprompters enable such good experimental design, this cannot be enforced. It is always possible for misuse of the optimization process to overfit.

Reviewer comment (Writing):
There are several typographic issues in the paper: Typo: GMS08K -> GSM8K (page 2, paragraph 4); Typo: format -> formats (page 4, Section 3.2); Typo: grammar: but -> or (page 5, Section 3.3); Typo: "As we assumes" in the example at the end of Section 3.3; Typo: these -> this, ensembles -> ensembling (page 6, stage 3); Typo: "specific math problems or particular LM." -> "specific to math problems or a particular LM." (page 7).

Author response:
Thank you for this extensive list of notes and typos. We believe we have now addressed all of these in the PDF. (We will update the submission abstract outside the PDF too, if the work is accepted. The system does not permit this change currently.)