M -Ped: Multi-Prompt Ensemble Decoding for Large Language Models (2025)

Abstract: With the widespread application of Large Language Models (LLMs) in the field of Natural Language Processing (NLP), enhancing their performance has become a research hotspot. This paper presents a novel multi-prompt ensemble decoding approach designed to bolster the generation quality of LLMs by leveraging the aggregation of outcomes from multiple prompts. Given a unique input X, we submit n variations of prompts with X to LLMs in batch mode to decode and derive probability distributions. For each token prediction, we calculate the ensemble probability by averaging the n probability distributions within the batch, utilizing this aggregated probability to generate the token. This technique is dubbed Inner-Batch Ensemble. To facilitate efficient batch inference, we implement a Left-Padding strategy to maintain uniform input lengths across the n prompts. Through extensive experimentation on diverse NLP tasks, including code generation, text simplification and machine translation, we demonstrate the efficacy of our method in enhancing LLM performance. The results show substantial improvements in pass@k rates, LENS metrics and BLEU scores over conventional methods.

## introduction large language models (llms) @xcite @xcite have demonstrated exceptional capabilities in understanding and generating natural language through extensive data pre-training, becoming the core driving force in the field of natural language processing (nlp). prompt technology @xcite @xcite @xcite @xcite , as a key to enhancing llms' performance, can strengthen the model's effects without altering model parameters, achieving seamless integration with downstream tasks. however, in the practical application of llms, the output results are closely related to the quality of the prompt, and a precise and effective prompt is crucial for improving the model's response quality. how to use prompts more efficiently to fully leverage the potential of llms has become a hot issue of common concern in academia and industry. ensemble decoding @xcite @xcite is a widely employed technique for enhancing the quality of model-generated outputs. typically, standard ensemble decoding refers to the process of combining the outputs of multiple distinct models on the same input. this approach, often termed model ensemble, leverages the diversity among models to reduce uncertainty and improve overall predictive performance. in theory, ensemble decoding could be applied to large language models (llms). however, llms already demand considerable memory resources, and implementing ensemble decoding with multiple llms presents significant challenges in terms of memory usage. in this paper, we introduce a particularly simple yet effective method: multi-prompt ensemble decoding (m-ped). this approach constructs n distinct prompts for a single query, generating n diverse input samples that are batched together and submitted to llms for inference. during the inference process, we average the prediction probabilities within the batch for each word prediction. to ensure batched inference is feasible, we specifically propose the use of left-padding technology to address the issue of varying prompt lengths within a batch. compared to traditional model ensemble methods, our approach shifts the focus of diversity from using different models to using different prompts. we have validated the effectiveness of this method across various tasks and multiple models, including extensive experiments on multiple test sets for code generation, text simplification and machine translation tasks. the re- our approach differs significantly from other multi-prompt methods, such as self-consistency @xcite and best-of-n @xcite . specifically, our method operates during the inference process, whereas other methods are applied after inference is completed. moreover, these alternative methods often require taskspecific evaluation metrics to be integrated into their post-inference processes. in contrast, our approach is metric-agnostic and does not rely on specific evaluation metrics. these other methods can be collectively categorized under minimum bayes risk (mbr) @xcite decoding. we provide an ablation study on mbr in the appendix. ## multi-prompt ensemble decoding the overall process of our proposed multi-prompt ensemble decoding is depicted in figure 1 . for a given distinct input @xmath0, x 2 , ..., x k } with a prompt p , we first generate a list of prompts {p 1 , p 2 , ..., p n }. then, we submit these n prompts and the input x to llms in batch for decoding to obtain probability distributions. we average the n probability distributions generated at the j-th position prediction within the batch to get the ensemble probability, and ultimately determine the output y 's y j . to ensure efficient batched inference is possible, we employ a left-padding strategy to ensure the lengths of the n inputs are consistent. ## main experiments we validated our approach across three tasks: code generation, text simplification, and machine translation. this experimental setup aligns with prior work on multi-prompts @xcite to ensure consistency. additionally, in the ablation study in appendix a, we analyzed the integration of this prior work and observed further improvements in performance. our primary experiments are based on two prompts. in the ablation study, we investigate the effects of using a larger number of prompts. for detailed settings regarding multi-prompt, please refer to appendix b.1. ## conclusions this study set out to address the challenge of enhancing the performance of llms in nlp tasks through the introduction of a multi-prompt ensemble decoding approach. our method, termed inner-batch ensemble, leverages the diversity of multiple prompts to aggregate their outcomes, thereby improving the generation quality of llms. the implementation of a left-padding strategy ensured efficient batch inference, allowing for uniform input lengths across various prompts. our extensive experiments across a range of nlp tasks-spanning code generation, text simplification and machine translation-demonstrated the effectiveness of our inner-batch ensemble method. the results were particularly compelling, with significant improvements observed in pass@k rates, lens metrics and bleu scores when compared to standard methods. the consistent enhancements across different tasks and metrics underscore the robustness and versatility of our approach.