Improving Minimum B ayes Risk Decoding with Multi-Prompt (2024)

Abstract: While instruction fine-tuned LLMs are effective text generators, sensitivity to prompt construction makes performance unstable and sub-optimal in practice. Relying on a single ‘best’ prompt cannot capture all differing approaches to a generation problem. Using this observation, we propose multi-prompt decoding, where many candidate generations are decoded from a prompt bank at inference-time. To ensemble candidates, we use Minimum Bayes Risk (MBR) decoding, which selects a final output using a trained value metric. We show multi-prompt improves MBR across a comprehensive set of conditional generation tasks, and show this is a result of estimating a more diverse and higher quality candidate space than that of a single prompt. Our experiments confirm multi-prompt improves generation across tasks, models and metrics.

## introduction minimum bayes risk (mbr) decoding @xcite improves the generation quality of large language models (llms) over standard, single-output decoding methods, such as beam search and sampling. mbr generates a set of candidates and selects the one with the highest expected utility, using all other hypotheses as references (see fig. 2 , left), following a simple intuition that a desirable output should be highly probable and consistent with others. mbr has been applied across a variety of nlp generation tasks @xcite @xcite . in particular, self-consistency @xcite , a special case of mbr, has become widely used to improve llm reasoning capabilities by ensembling reasoning paths. a central question to improve the generation quality of mbr decoding is how to balance between diversity and adequacy within the candidate set. prior work has found success using samplingbased decoding to generate diverse hypotheses @xcite @xcite . however, naively increasing the sampling temperature eventually degrades the quality of the candidates. recently, instruction fine-tuned llms @xcite have opened up the possibility of writing prompts in various formats to elicit higher diversity generations. as these models are observed to be sensitive to prompt design, a slight change in phrasing or the inclusion of more relevant example can significantly impact model behavior @xcite . taking advantage of the prompt sensitivity of llms, we introduce multi-prompt mbr decoding, which samples candidates using a bank of humanor model-written prompts (see figure 2 , right). intuitively, exploring a variety of prompts enables the generation of diverse, high quality hypotheses that provide a closer representation of the true output distribution. by guiding the model towards different regions of the output space, each prompt captures unique sequences that are coherent and relevant to the specific input example. we experiment with three distinct generation tasks: text simplification @xcite , machine translation @xcite , and code generation @xcite . each task assess the impact of different prompt components on multiprompt mbr, such as instance-level prompts for code, task descriptions for simplification, and incontext examples for translation. to account for the relative quality between prompts, we develop different strategies for selecting prompts that outperform a baseline random choice: sampling prompts from a large prompt bank based on their usage on an unlabeled set of task data and selecting prompts using embedding-based heuristics without any examples. we evaluate multi-prompt mbr on a broad range of llms including open-source models such as llama 2 @xcite and state-of-theart closed-source models such as @xcite . our results show multi-prompt mbr consistently improves single-prompt mbr across all three tasks and model scales, with gains of up to 7% on humaneval @xcite and 5 points of lens score on simpeval @xcite . figure 1 displays results for models at the 7b scale. finally, we study the dynamics between different utility and evaluation metrics, revealing that multi-prompt mbr with one metric improves performance universally across metrics. ## preliminaries instruction fine-tuned llms are trained to follow arbitrary natural language task descriptions @xcite . given an input x and prompt ρ, an autoregressive language model π θ parameterized by θ estimates an output sequence y ∼ π θ (x, ρ) using an decoding algorithm by sampling the next token conditioned on the input π θ (y i |@xmath0, x, ρ). the decoding algorithm aims to generate y by maximizing the sequence likelihood over the language model distribution π θ (y|x, ρ) = π t @xmath1, x, ρ). minimum bayes risk decoding. in practice, the highest likelihood sequence does not necessarily yield the highest quality generation @xcite . from this observation, mbr decoding @xcite first samples a set of hypotheses h from the model π θ , approximating the true distribution of output space y, then selects the output ŷmbr that maximizes the expected utility (or minimizes the expected loss in traditional formulation) with respect to a set of references r: you are an artificial intelligence designed to simplify human written i would like you to simplify the following sentence such that the write a simpler version such that a non-english speaker or an individu... multi-prompt instruction fine-tuned llm candidate selection e.g., gpt-4, llama chat, alma with trained value metric bertscore, lens, comet rewrite the following complex sentence in order to make it simple ... single prompt figure 2: multi-prompt mbr generates candidates using a human-or model-written prompt bank and selects the highest pairwise score with a trained value metric. ## multi-prompt mbr decoding prior work on mbr decoding primarily uses models trained or fine-tuned for a specific generation task @xcite . with instruction fine-tuned llms, the input x is contained within a structured prompt ρ, consisting of task instruction and/or in-context examples. earlier studies have extensively documented that the design of the prompt has a dramatic impact on overall performance @xcite @xcite . to investigate this phenomenon, we show in figure 3a (bottom) the likelihoods and quality of samples from 10 prompts of varying performance for a text simplification task, measuring quality as the lens metric score against a set of gold references. greedy sampling (τ = 0) estimates different sequences for each instruction, with single prompt (figure 3a , top) generating a single sequence. as we increase temperature τ , generations from a single prompt simply exhibit noise centered around the mode of the highest likelihood sequence, while multi-prompt estimates a generations around modes uniquely defined by each prompt. for instance, one of the prompts (i.e., prompt 9 highlighted in green) produces the highest quality generation for this one input sentence, despite having a low performance over the entire dataset. in fact, no prompt consistently produces the highest quality sequences, as illustrated in figure 3b , rather prompts are most effective at different inputs. building upon these insights, we propose multiprompt mbr decoding, depicted in figure 2 , where the mbr hypothesis set h consists of outputs sampled from n distinct prompts ρ: @xcite show that mbr seeks the mode of some distribution q over a quality feature ϕ(y) applied to the output space rather than the mode of the model's distribution: we hypothesize, in expectation, the mode of ϕ(y) across outputs from multiple prompts has higher downstream performance compared to that derived from a single prompt. this is empirically supported by our example, where figure 3c shows that multi-prompt mbr outperforms individual singleprompt mbr across the full task dataset. although multi-prompt ensembles hypothesis spaces between prompts, some notion of objective quality still exists when constructing the prompt bank. as shown in figure 3c , the majority of the 10 human-written prompts fall within a 10-point range of lens scores when evaluated on the task dataset but a few prompts consistently produce low-quality generation. therefore, to account for the hierarchy in prompt quality, we propose two methods for choosing the prompts used at generation time from a prompt bank p: sampling from a learned distribution of prompts, based on a small unlabeled train set ( §3.1); and selecting a subset of prompts based on heuristics in the absence of a train set ( §3.2). ## experiment setup in this section, we describe the experimental details for evaluating the efficacy of multi-prompt mbr decoding across tasks, prompt setups, models, and utility metrics, with results and analyses in §5. ## experiment results we compare multi-prompt decoding to traditional mbr ( §5.1), ablate the prompt sampling mechanism ( §5.2), vary model architectures ( §5.3), evaluate across utility metrics ( §5.4) and finally evaluate multi-prompt on efficient mbr alternatives ( §5.5). ## conclusion in this work, we propose multi-prompt, a generalized case of mbr for conditional text generation. multi-prompt successfully ensembles outputs of instruction fine-tuned language models across prompt constructions and in-context examples. we highlight the importance of prompt selection and sampling when constructing the prompt bank with top-p prompt sampling and further verify our results across tasks, models and utility metrics.