Paper: Prompt Sketching for Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
Most of the explored tasks, including logical reasoning, question answering, and arithmetic reasoning, use the *multi-variable* prompting method (BeamVar, Var) as the sketch (Figure 3), which is actually a variant of the self-consistency [1] method: sample multiple chain-of-thoughts and then aggregate. Hence a fair comparison should be between the proposed method and self-consistency-based chain-of-thought, under the exact same number of samples.

Author response:
**Q: How does Sketching with BeamVar/Var relate to Self-Consistency and can you add it as a baseline?**  
The self-consistency [1] decoding strategy improves LLM reasoning by greedily sampling multiple (chain-of-thought) reasoning paths and aggregating the resulting final answers, by (unweighted or weighted) majority vote. In contrast, sketch-aware decoders generate answers token-by-token or thought-by-thought (not path by path), can branch on the top-n alternatives at each step, and do not aggregate the final results. Instead, a single sequence is chosen as the one answer to an input, considering the model likelihood of the underlying token sequence only (no aggregation or majority voting). With this, self-consistency and sketching can be considered orthogonal approaches and could even be applied jointly, i.e., sample-decode a sketch multiple times and aggregate the results.  
We added a comparison of sketching and self-consistency to the revised manuscript (Appendix D). For this comparison, we choose $n=2$/$n=4$ samples for self-consistency, as this aligns with the computational overhead of our branching decoders BeamVar/Var with $n=2$. However, our results show that, in many cases, self-consistency with such a low number of samples does not even outperform argmax CoT which is the main baseline we outcompete with sketching. This aligns with the self-consistency paper itself [1], which reports that around 5-20 consistency samples are required to outperform simple argmax CoT. We thus find that within this compute budget ($n=2$/$n=4$), argmax CoT and thus sketching both outperform self-consistency decoding.

Reviewer comment (Novelty):
The novelty of the proposed method compared to self-consistency should be discussed in detail in this paper.

Author response:
**Q: How does Sketching with BeamVar/Var relate to Self-Consistency and can you add it as a baseline?**  
The self-consistency [1] decoding strategy improves LLM reasoning by greedily sampling multiple (chain-of-thought) reasoning paths and aggregating the resulting final answers, by (unweighted or weighted) majority vote. In contrast, sketch-aware decoders generate answers token-by-token or thought-by-thought (not path by path), can branch on the top-n alternatives at each step, and do not aggregate the final results. Instead, a single sequence is chosen as the one answer to an input, considering the model likelihood of the underlying token sequence only (no aggregation or majority voting). With this, self-consistency and sketching can be considered orthogonal approaches and could even be applied jointly, i.e., sample-decode a sketch multiple times and aggregate the results.  
We added a comparison of sketching and self-consistency to the revised manuscript (Appendix D). For this comparison, we choose $n=2$/$n=4$ samples for self-consistency, as this aligns with the computational overhead of our branching decoders BeamVar/Var with $n=2$. However, our results show that, in many cases, self-consistency with such a low number of samples does not even outperform argmax CoT which is the main baseline we outcompete with sketching. This aligns with the self-consistency paper itself [1], which reports that around 5-20 consistency samples are required to outperform simple argmax CoT. We thus find that within this compute budget ($n=2$/$n=4$), argmax CoT and thus sketching both outperform self-consistency decoding.

Reviewer comment (Experiments):
Can the authors add self-consistency with the same number of samples as a baseline?

Author response:
**Q: How does Sketching with BeamVar/Var relate to Self-Consistency and can you add it as a baseline?**  
The self-consistency [1] decoding strategy improves LLM reasoning by greedily sampling multiple (chain-of-thought) reasoning paths and aggregating the resulting final answers, by (unweighted or weighted) majority vote. In contrast, sketch-aware decoders generate answers token-by-token or thought-by-thought (not path by path), can branch on the top-n alternatives at each step, and do not aggregate the final results. Instead, a single sequence is chosen as the one answer to an input, considering the model likelihood of the underlying token sequence only (no aggregation or majority voting). With this, self-consistency and sketching can be considered orthogonal approaches and could even be applied jointly, i.e., sample-decode a sketch multiple times and aggregate the results.  
We added a comparison of sketching and self-consistency to the revised manuscript (Appendix D). For this comparison, we choose $n=2$/$n=4$ samples for self-consistency, as this aligns with the computational overhead of our branching decoders BeamVar/Var with $n=2$. However, our results show that, in many cases, self-consistency with such a low number of samples does not even outperform argmax CoT which is the main baseline we outcompete with sketching. This aligns with the self-consistency paper itself [1], which reports that around 5-20 consistency samples are required to outperform simple argmax CoT. We thus find that within this compute budget ($n=2$/$n=4$), argmax CoT and thus sketching both outperform self-consistency decoding.

Reviewer comment (Experiments):
In section 4.2, some novel tasks are explored and could potentially show the benefits of the proposed sketching. However, the experiments are extremely small-scale (10 Sudoku puzzles, 10 Dungeon environments), so it is unclear whether the proposed method indeed outperforms existing methods.

Author response:
**Q: Can you extend your evaluation of additional applications and further demonstrate the effectiveness of sketching in this context?**  
We have scaled our experiments on the additional applications (Section 4.2) by a factor of 10 and now evaluate a larger number of samples (100 samples per model/decoder configuration). We report the results in the updated manuscript. Overall, we observe comparable trends to before, i.e., sketch-aware decoders clearly show the ability to backtrack in Sudoku solving and solve the Dungeon Escape tasks more efficiently (fewer steps), when compared to naive argmax decoding. We have also added an additional experiment on TextWorld [3] exploration, which leverages our decoders for LLM-guided world exploration. Again, we find that sketch-aware decoders clearly outperform greedy argmax decoding, completing TextWorld quests of different lengths with 20% fewer steps. We further discuss our TextWorld results in Appendix B.2.

Reviewer comment (Evaluation):
From Table 6, the confidence intervals are fairly large, and it is unclear which method is significantly better compared to the others. Can the authors clarify which result is statistically significant?

Author response:
**Q: Can you clarify which results are statistically significant given the evaluation sample sizes and  fairly large confidence intervals in Table 6?**  
We understand your concerns regarding sample size and statistical significance for the OpenAI-specific part of our evaluation. Unfortunately, we cannot scale our OpenAI experiments further, due to cost constraints ($4000 for the results at the current scale). However, we have expanded our Llama-2-based experiments (Table 2) to include all decoder/dataset configurations and evaluate 10 times as many samples as in the OpenAI experiments (1000 per dataset, or the full datasets in many cases) and also provide confidence intervals for the Llama results in the new appendix Table 8. In this much more scaled Llama-2 setting, we observe comparable trends to the smaller-scale OpenAI experiments: Across tasks, sketching improves overall reasoning performance when compared to sequential prompting, while sketch-aware decoders can provide further gains in exchange for higher computational cost.

Reviewer comment (Reproducibility):
Can the authors discuss in more detail the exact computational cost used for the proposed method?

Author response:
**Q: Can you discuss the computational cost of the proposed method in further detail?**  
Computationally, argmax sketching is cheaper than sequential decoding for the same output. This is because in sketching the model does not have to autoregressively process fixed template parts (assuming sufficient model access and key-value caching). For BeamVar the computational cost is increased by a factor of beam width $n$, whereas Var requires a factor of $n^2$ more sequences to be explored (2x and 4x, respectively, for our main evaluation). This is a considerable computational overhead, but, as known from traditional beam search, can be worth the additional compute if downstream performance is absolutely crucial. It is also more cost-effective than, e.g., self-consistency decoding, which requires 5-20 consistency samples, resulting in a 5-20x overhead. In practice, this kind of performance/accuracy trade-off of course depends on the concrete use case, compute budget, and user requirements.

Reviewer comment (Evaluation):
Generating templates requires human intervention and may necessitate significant efforts until finding a template working well. Also, potentially, templates can overfit to evaluation datasets.

Author response:
**Q: Can you comment on the overhead of writing sketches as opposed to prompts?**  
Similar to prompt engineering, templates have to be constructed, tested, and debugged with example data to ensure the best performance. With current models, however, this prompt/template engineering effort cannot be avoided, as a single input/sentence is typically not sufficient to ensure that the model conforms to very specific and precise requirements. In this context, we consider sketching a more controlled form of prompting, where the output format is guaranteed and not up to prompt engineering and randomness in the generation process. Nonetheless, we agree that a human effort has to be made to properly prompt, template, and use LLMs. See also Section 4.3 for more discussion of this aspect.

Reviewer comment (Evaluation):
It does not work well for small LMs.

Author response:
**Q: Is prompt-sketching less effective with smaller LLMs?**  
We note that even with a smaller Llama-2 Chat model (13B parameters), sketching outperforms sequential prompting on 6/8 reasoning tasks while exhibiting even greater performance gains than for large OpenAI models (up to 8% vs. up to 4%). However, as we acknowledge in our evaluation, Llama-2 seems incapable of solving the Matrix Shape and AQuA reasoning tasks, irrespective of the prompting/sketching scheme, leading to inconclusive results on these datasets. Other tasks are not as strongly affected and show comparable trends to the OpenAI results. Further, we demonstrate in Appendix B.1 that sketching can be very effective with smaller models, for instance when asking a model to produce valid JSON output: For the small text-curie-001, sketching allows us to guarantee valid JSON output, whereas the same model with a corresponding prompts, is unable to produce any valid JSON at all. 

Potentially, our description in Section 4 was unclear in this regard. We have updated it in the rebuttal revision to make the above points more clear.

Reviewer comment (Evaluation):
Evaluation results are given with limited amounts of data, which may harm the credibility of the results. Especially, confidence intervals in Table 6 look pretty large.

Author response:
**Q: Does the sample size in the evaluation impact the credibility of the results?**  
We understand your concerns regarding sample size and statistical significance for the OpenAI-specific part of our evaluation. Unfortunately, we cannot scale our OpenAI experiments further, due to cost constraints ($4000 for the results at the current scale). However, we have expanded our Llama-2-based experiments (Table 2) to include all decoder/dataset configurations and for 10 times as many samples as in the OpenAI experiments (1000 per dataset, or the full datasets in many cases). We also provide confidence intervals for the Llama results in the new appendix Table 8. In this much larger-scale Llama-2 setting, we observe comparable trends to the smaller-scale OpenAI experiment: Across tasks, sketching improves overall reasoning performance when compared to sequential prompting, while sketch-aware decoders can provide further gains in exchange for higher computational cost.

Reviewer comment (Novelty):
Most of new applications look already doable by guidance-ai (https://github.com/guidance-ai/guidance ), which is cited in the paper. Also, naive stop-and-go is not compared in main results.

Author response:
**Q: How does the presented approach differ from Guidance?**  
Indeed, practical tools like Guidance also enable a form of stop-and-go inference, which can be seen as a simple form of sketching. However, Guidance is an ad-hoc approach, where templates are simply decoded using multiple isolated LLM calls. In contrast, sketching theoretically anchors this approach as a multi-part sequence decoding problem, enabling sketch-aware decoders. Further, prior work (including Guidance) does not provide insights on whether this form of structural guidance positively impacts model reasoning capabilities. This is evaluated as the argmax version of prompt sketching. Our experimental results are thus also of interest to Guidance-like frameworks, as we show that structural guidance during inference can indeed improve reasoning capabilities, beyond syntax and structure. To the best of our knowledge, this has not been shown before and therefore improves the understanding of simple stop-and-go inference in general. We have updated the introduction of the paper to better reflect this.

Reviewer comment (Evaluation):
The experiments show that stop-and-go inference works well, and the proposed method does not significantly improve performance despite the additional overhead.

Author response:
**Q: Can you comment on the effectiveness of sketching/sketch-aware decoders with different tasks and models, and how it compares to argmax+CoT?**  
We consider the contribution of our experimental results to be twofold:
* We demonstrate that template-guided inference via simple argmax-decoded sketches (stop-and-go) and comparatively simple structural templates can already improve task accuracy on reasoning tasks (OpenAI: 7/8, Llama-2: 6/8) significantly (4% and 8% respectively). To the best of our knowledge, this has not been demonstrated before, as existing work on template-guided inference does not discuss the impact on reasoning performance. We thus, for the first time, empirically validate the effectiveness of template-guided (stop-and-go) inference on reasoning tasks, using our sketching framework. Computationally, argmax sketching is cheaper than sequential decoding for the same output (e.g., argmax+CoT), as in sketching/stop-and-go, the model does not have to autoregressively process fixed template parts (assuming sufficient model access and key-value caching). In comparison, simple sketching thus provides accuracy improvements without necessarily incurring any computational overhead (potentially even reducing it).
* We also demonstrate that while argmax sketching/stop-and-go inference can be effective, sketch-aware decoding brings even further improvements. These decoders incur computational overhead, but, as with traditional beam search, can be worth the additional compute if downstream performance is absolutely crucial. Of course, in practice this trade-off depends entirely on the concrete use case, compute budget, and user requirements.

Reviewer comment (Experiments):
On many of the tasks, simple autoregressive CoT seems sufficiently close in performance.

Author response:
**Q: Can you comment on the effectiveness of sketching/sketch-aware decoders with different tasks and models, and how it compares to argmax+CoT?**  
We consider the contribution of our experimental results to be twofold:
* We demonstrate that template-guided inference via simple argmax-decoded sketches (stop-and-go) and comparatively simple structural templates can already improve task accuracy on reasoning tasks (OpenAI: 7/8, Llama-2: 6/8) significantly (4% and 8% respectively). To the best of our knowledge, this has not been demonstrated before, as existing work on template-guided inference does not discuss the impact on reasoning performance. We thus, for the first time, empirically validate the effectiveness of template-guided (stop-and-go) inference on reasoning tasks, using our sketching framework. Computationally, argmax sketching is cheaper than sequential decoding for the same output (e.g., argmax+CoT), as in sketching/stop-and-go, the model does not have to autoregressively process fixed template parts (assuming sufficient model access and key-value caching). In comparison, simple sketching thus provides accuracy improvements without necessarily incurring any computational overhead (potentially even reducing it).
* We also demonstrate that while argmax sketching/stop-and-go inference can be effective, sketch-aware decoding brings even further improvements. These decoders incur computational overhead, but, as with traditional beam search, can be worth the additional compute if downstream performance is absolutely crucial. Of course, in practice this trade-off depends entirely on the concrete use case, compute budget, and user requirements.

Reviewer comment (Evaluation):
While the paper provides some additional applications for prompt sketches, the tasks and the performance on the tasks are not entirely convincing.

Author response:
**Q: Can you extend your evaluation of additional applications and further demonstrate the effectiveness of sketching in this context?**  
We have scaled our experiments regarding additional applications by a factor of 10, and now evaluate a larger number of samples (100 samples per model/decoder configuration). We report the results in the updated manuscript in Table 3. Overall, we observe comparable trends to before, i.e., sketch-aware decoders clearly show the ability to backtrack in Sudoku solving and solve the Dungeon Escape tasks more efficiently (fewer steps). We have also added an additional experiment on TextWorld [1] exploration (Table 3), which leverages our decoders for LLM-guided world exploration. Again, we find that sketch-aware decoders clearly outperform greedy argmax decoding, completing TextWorld quests of different lengths with ~20% fewer steps. We discuss our TextWorld result in appendix B.2.

Reviewer comment (Reproducibility):
How is the custom decoding applied when using OpenAI API?

Author response:
**Q: How is custom decoding possible with the OpenAI API?**  
The proposed decoders only require rather minimal support by the API/model. In particular, we require the possibility to extend all of the sequences in the beam and obtain logprobs for these sequences for at least their top $n$ continuations. For the OpenAI Completion API models we use in the paper, these capabilities are available.

Reviewer comment (Experiments):
I'm curious about the results if few-shot prompts are used.

Author response:
**Q: How does sketching perform when combined with few-shot prompting?**  
Sketching and few-shot prompting are orthogonal strategies, i.e., can be applied independently and jointly. We provide results on combining both in Appendix C.1. Overall, we find that few-shot prompting improves performance across the board, however, sketching still outcompetes sequential prompting, and in some cases zero-shot sketching even outperforms two-shot sequential prompting, suggesting that sketching can even serve as a replacement for few-shot demonstrations.

Reviewer comment (Experiments):
I couldn't quite follow the motivation: what problem with existing decoding techniques is prompt sketching meant to address? Figure 1 comes closest to illustrating the problem, but it was not particularly compelling. (I am not sure which model was used to generate Figure 1, but I tried copying the prompt and constraint into text-davinci-003 and it had no trouble following the constraint.) To be sure, there are many sketches that I am sure GPT-3 would often fail to follow, even if the sketch were included in the prompt; you can encode arbitrarily difficult infilling problems into sketches. But the sketches presented in this paper are enforcing very simple formatting constraints on, e.g., the list of thoughts generated for a chain-of-thought prompt. What failure modes do you see when just explaining to the model that it should follow the desired format (e.g. by pasting the sketch into the context)? Do failures to follow the format cause failures in reasoning? How exactly do VAR and BEAM_VAR address these failures? (Can they really be doing much, at a beam size of only n=2?)

Author response:
**Q: Can you clarify the motivation for sketching and provide failure modes of existing models/decoding methods?**  
As pointed out by the reviewer, template-guided decoding can also be implemented by simply showing the LLM a template and asking it to produce a corresponding response. However, there are several limitations and failure modes with this approach:
- *Instruction Following Capabilities:* Only big and powerful models are currently capable of this kind of instruction-following (e.g., see how `text-curie-001` fails for this kind of task in Table 3, Sudoku). In contrast, sketching also works with small models, which may not understand the template, but can still produce useful completions for individual template variables.
- *Prompt Size:* Including large templates fully in the prompt will greatly increase the number of required tokens, possibly even exceeding the context window of the model. Further, LLM retrieval across large inputs remains difficult, which can lead to deviations from the provided template. In contrast, sketching guides the model through the template, step by step, without increasing the total number of tokens or requiring long-range retrieval.
- *Accuracy, Reliability, and Automation:* Instruction-following models are still not perfect, which means templates may be violated and not followed strictly. In contrast, sketching provides a 100% guarantee that the template will be adhered to. This is crucial in automated settings, where the LLM’s output format must always be correct to enable processing by other systems that expect a specific format (e.g., function calling).
- *Dynamic Templates and Interactive Environment:* In some cases, providing the full template may also not be possible, as it may not be known fully ahead of time. For instance, templates may be dynamic, e.g., multiple repetitions of some template part, until a certain variable value is produced. Models can be instructed to respect such constraints, but this kind of prompting ultimately remains unreliable while sketching can enforce these properties strictly (cf. sketched CoT in the paper). Further, sketching is also applicable to interactive settings, where the template depends on and reacts to model output. For instance, in Table 3 we experimented with Dungeon-style graph traversal (Table 3), and in the updated revision, we added an additional experiment on TextWorld exploration. 

**Q: How exactly do BeamVar/Var address these failure modes (with $n=2$)?**  
As pointed out above, sketching alone already greatly alleviates these failure modes as it leads to strict adherence to the template and allows for scripted follow-up. Sketch-aware decoders (BeamVar/Var) additionally allow us to jointly score multiple variables (while standard decoders only allow the last variable to be scored conditioned on all preceding ones). This enables picking variables based on later sequences in the template, which is particularly important in dynamic environments where the template text depends on the value of prior variables. While modern RLHF models are very strong and typically assign high probabilities to the top-1 token, a small beam width of n=2, can already be useful to track an alternative hypothesis when ambiguities arise (demonstrated by our experiments on reasoning tasks). For our more advanced experiments with other applications (cf. Table 3), we apply BeamVar and Var with higher beam width (n=5 and n=3 respectively), as clarified in the revised draft, which enables the exploration of a broader hypotheses space.

**Q: Why are you focusing on relatively simple formatting constraints?**  
We chose to focus on simple sketches, to examine whether structural support can improve model reasoning capabilities at all. Indeed, more complex or even grammar-guided approaches are possible, however, the design space of such sketches/reasoning grammars is very large and requires significant prompt engineering. In contrast, the main goal of this research is to establish foundational insight on whether structural guidance of the LLM reasoning process can improve its capabilities at all. Nonetheless, our sketching framework and decoders are general and capable of more complicated forms of template-guided inference, setting up future work to explore this direction.

Reviewer comment (Experiments):
The experiments provide somewhat weak evidence for the value of the new decoding methods. In different tasks, it often seems to be the case that *one* of the methods outperforms an argmax baseline, whereas the *other* method underperforms the baseline, and which method wins varies from task to task. Even when the new decoding methods provide a modest advantage over argmax decoding, it is not clear whether the advantage is worth the added computational cost (or dollar cost, for OpenAI models).

Author response:
**Q: Can you comment on the effectiveness of your methods with different tasks and models? Is it worth the computational overhead?**  
We consider the contribution of our experimental results to be twofold:
* We demonstrate that template-guided inference via simple argmax-decoded sketches and comparatively simple structural templates can already improve task accuracy on reasoning tasks (OpenAI: $7/8$, Llama-2: $6/8$) greatly ($4%$ and $8%$ respectively). To the best of our knowledge, this has not been demonstrated before, as existing work on template-guided inference does not focus on the impact on reasoning performance. Computationally, argmax sketching is also cheaper than sequential decoding for the same output, as in sketching, the model does not have to autoregressively process fixed template parts (assuming sufficient model access and key-value caching).
* We also demonstrate that while argmax sketching/stop-and-go inference can be effective, sketch-aware decoding brings even further improvements. For Beam-Var the computational cost is increased by a factor of beam width $n$, whereas Var requires a factor of $n^2$ more sequences to be explored. This is a considerable computational overhead, but, as known from traditional beam search, can be worth the additional compute if downstream performance is absolutely crucial. 

More broadly, we mostly find VAR to be particularly effective with interleaved reasoning whereas BeamVar prevails in multiple-choice and math questions.

Reviewer comment (Experiments):
I am not convinced the experimental setup is completely debugged. For example, in chain-of-thought for the "date understanding" task, a period is used as the stopping phrase for each thought. However, periods show up frequently in dates (e.g., "Sept. 1"), and this stopping-phrase is clearly causing the model to cut off thoughts early (page 21). Some experimental settings are also missing details; e.g., in the single-variable chain-of-thought prompts, it is unclear when the [COT] variable ends -- I did not see a stopping phrase reported.

Author response:
**Q: Does stopping on “.” with sketched chain-of-thought prompts restrict the model's reasoning? If no stopping phrase is specified, when does a variable end?**   
Thank you for this close attention to detail. Indeed, a badly chosen stopping phrase can impact decoding in this way, just like a badly constructed prompt will impact performance. However, upon manual inspection, we don’t find this to be a particular issue in this case. Especially in chain-of-thought prompts, stopping on “.” mostly serves as a proxy for structuring the LLMs response, where LLMs will also recover and continue potentially truncated “thoughts”. Generally, we constructed sketches just like prompts, by developing, testing, and debugging them using real example data (also discussed in 4.3). If no stopping phrase is provided (e.g., for CoT), our decoders continue generation until the model emits the end-of-sequence token on its own.

Reviewer comment (Theory):
Some of the algorithmic choices in VAR / BEAM_VAR were not sufficiently justified, and struck me as slightly odd. For example, the VAR algorithm shrinks the n^2 generations for a variable back down to a beam width of n *before* adding the next deterministic chunk. But I thought a key point of these algorithms was to enable the next deterministic chunk to provide a "score" for the proposed variable values; wouldn't it make more sense to rank all n^2 variable proposals by how well they fit with the next deterministic chunk, scale back down to n, and then generate proposals for the next variable?

Author response:
**Q: Why do you shrink VARs $n^2$ generations back down to beam width $n$, before adding the next deterministic chunk rather than extending all $n^2$?**  
This is an interesting proposal. The reason for our choice is mostly based on cost considerations w.r.t. the OpenAI API. Each additional scored sequence incurs an additional request at the full length of the current sequence with OpenAI. Since `top_n` still maintains the highest scoring continuation, we reduce this cost by focusing on the “most promising” sequences in the current set.  With open/accessible models, however, the proposed alternative would definitely be sensible, considering that scoring a sequence with standard transformers corresponds to a single forward pass only. In our published open source decoder library dclib, this can be easily implemented by moving a single line.