Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis (2024)

Abstract: Previous work in prompt engineering for large language models has introduced different gradient-free probability-based prompt selection methods that aim to choose the optimal prompt among the candidates for a given task but have failed to provide a comprehensive and fair comparison between each other. In this paper, we propose a unified framework to interpret and evaluate the existing probability-based prompt selection methods by performing extensive experiments on 13 common and diverse NLP tasks. We find that each of the existing methods can be interpreted as some variant of the method that maximizes mutual information between the input and the predicted output (MI). Utilizing this finding, we develop several other combinatorial variants of MI and increase the effectiveness of the oracle prompt selection method from 87.79% to 94.98%, measured as the ratio of the performance of the selected prompt to that of the optimal oracle prompt. Furthermore, considering that all the methods rely on the output probability distribution of the model that might be biased, we propose a novel calibration method called Calibration by Marginalization (CBM) that is orthogonal to the existing methods and helps increase the prompt selection effectiveness of the best method to 96.85%, achieving 99.44% of the oracle prompt F1 without calibration.1

## introduction large language models (llms) have demonstrated remarkable performance in solving various natural language processing tasks through prompt-based learning without requiring additional task-specific training (brown et al., 2020; dong et al., 2023). however, the performance of llms can heavily fluctuate according to the choice of prompts (zhao et al., 2021; holtzman et al., 2021; lu et al., 2022). while various prompt engineering approaches have been proposed to mitigate this issue, the nontrivial prerequisites of many of these methods, such as training an additional model and/or using an additional component, have been a bottleneck to their real application (liu et al., 2023; li and liang, 2021; jiang et al., 2020; prasad et al., 2023; liu et al., 2022; rubin et al., 2022). on the other hand, probability-based prompt selection methods do not require any additional parameter updates or additional components foot_0 and thus provide a promising and easily applicable solution; these methods aim to select the prompt from a set of prompts that is expected to be most effective in helping a language model to make correct predictions solely based on the probability distribution of the model (sorensen et al., 2022; lu et al., 2022; wu et al., 2023; liao et al., 2022; gonen et al., 2023). however, despite their ease of utilization, there has been a lack of comprehensive comparative evaluation between existing probability-based prompt selection methods, as each method is proposed in different setups and evaluated on different datasets, evaluation instances, sets of prompts, and models. in this paper, we first carefully design a unified evaluation setup to facilitate a fair comparison between different prompt selection methods. our unified evaluation reveals that no single method consistently outperforms other methods across all datasets and that figure 1 : (a) f1 of the prompts selected by different probability-based prompt selection methods, averaged across 13 datasets. per-dataset f1 and accuracy are shown in figure 9 . the methods without super/subscripts are the existing methods ( furthermore, we find the need for a better approximation of the llm's output probability distribution, considering that all probability-based prompt selection methods rely on the probabilistic estimates from the model that might be biased. therefore, by drawing a connection between the existing model output probability calibration methods (zhao et al., 2021; holtzman et al., 2021), we propose an enhanced calibration method, calibration by marginalization (cbm). cbm significantly improves the prompt selection performance of several methods when applied to calibrate the output probability of llms, increasing the best-scaled f1 to 96.85% (mi @xcite a of figure 1a ), achieving 99.44% of the oracle prompt f1 under the uncalibrated scenario. cbm also proves to show the most robust answer selection enhancement across multiple datasets compared to the existing calibration methods (figure 1b ). ## probability-based prompt selection in this section, we perform a unified evaluation of existing probability-based prompt selection methods. first, we describe the task of probability-based prompt selection in section 2.1. next, we briefly introduce each of the existing methods in section 2.2. then, we describe our experimental setup for unified evaluation in section 2.3 and present the evaluation results in section 2.4. ## improving mi via unified analysis in this section, we first derive a unified view of prompt selection methods in section 3.1 and show that each method other than mi roughly corresponds to a sub-term of the equation of mi and revisit the previous experimental results for a unified analysis in section 3.2. then, from the unified view and analysis, we identify the differences between methods, particularly mi, ge, and mdl, and derive a few combinational variants by transferring design elements across methods which improves the prompt selection performance of mi. ## improving prompt selection through enhanced probability calibration while the previous section enhances prompt selection performance using combinatorial variants, in this section, we explore an orthogonal approach to further improve prompt selection: model output probability calibration. since all the prompt selection methods except for ppl depend on the model output probability @xcite to calculate prompt selection score (pss), the stability and reliability of @xcite affect their prompt selection performance. however, previous works have pointed out that @xcite is unstable without calibration. @xcite to address the issue, zhao et al. (2021) suggest contextual calibration (cc), which reduces bias towards each answer choice by employing content-free inputs (''n/a'', ''[mask]'', ''''), while holtzman et al. (2021) present domain conditional pointwise mutual information (pmi dc ) by reweighting each answer choice based on its task-specific prior likelihood. we summarize the two methods for answer selection in one might assume that these existing calibration methods would effectively calibrate @xcite for pss. however, through the experiments described in section 4.1, we reveal in section 4.2 the results that these methods have limitations for prompt selection and even answer selection across numerous datasets. in response, we propose an enhanced calibration method, calibration by marginalization (cbm), in section 4.3. section 4.4 shows that cbm notably improves prompt selection for most methods, particularly mi and mdl m , enabling them to achieve the highest prompt selection performance compared to all other methods. furthermore, cbm's answer selection enhancement is the most robust across various datasets when compared to existing calibration methods. ## discussion in this section, we discuss various findings that are relevant to our main experiments. figure 7a shows that the effectiveness of a probability-based prompt selection method remains consistent across models of different types and numbers of parameters, justifying our choice of using a single model (opt 2.7b) as the representative for all experiments. figure 7b shows that the trend of correlation between prompt selection score and performance of the selected prompt is also quite consistent between different models. figure 8 shows the mean and standard deviation of the result of prompt selection among five different subsets of 50 prompts randomly sampled from the full set of 100 prompts, using the mainly discussed methods. the result shows that the performance of instance-wise prompt selection methods (mi agl , mi al , mdl) is not stable, likely due to the noisy nature of selecting one prompt for each instance. however, the performance of mi @xcite a and mdl @xcite m still achieves the highest performance and also shows the lowest standard deviation, proving the effectiveness of cbm. through additional analysis, we find that (1) while strong performance in prompt selection does not consistently correlate with prompt selection score, a broadly positive correlation is observed when averaged across most methods; (2) cbm improves the performance of mdl m by mitigating overconfidence; (3) mi, ge, and cbm methods face limitations when applied to dynamic datasets with extreme label imbalance; (4) top-performing prompt selection methods from the zero-shot setting, like mi @xcite a and mdl @xcite m , retain their effectiveness in the few-shot setting, further validating their robustness across different conditions. ## related works recent advances in llms have created the paradigm of prompt-based learning, which gives the benefit that a single pretrained llm can be used to solve a great number of tasks with task-specific prompts. however, the performance of llms can heavily fluctuate according to the choice of prompts (zhao et al., 2021; holtzman et al., 2021; lu et al., 2022). to mitigate this issue, prompt engineering attempts to find the prompt that results in the most effective performance on the downstream task (liu et al., 2023). automatic prompt engineering methods can be largely divided into two groups: the methods that use discrete prompts where the prompts are human-understandable actual text strings, and the methods that optimize continuous prompts where the prompts lie in the embedding space of the model (li and liang, 2021; shin et al., 2020). probability-based prompt selection methods that we study in this work (section 2.2) fall into the former group; most of the methods of the latter group require gradient-based training, while probability-based prompt selection does not perform any gradient-based update. prompt engineering methods using discrete prompts include prompt paraphrasing, prompt generation, and prompt selection. among these, prompt paraphrasing or generation approaches can be used together with probability-based selection methods; prompt selection can be performed on the prompts generated through prompt paraphrasing or generation (jiang et al., 2020; mishra et al., 2022; gao et al., 2021; wang et al., 2023; prasad et al., 2023; kim et al., 2022; deng et al., 2022). among prompt selection methods other than the probability-based approaches, a large portion of the methods are not easily utilizable since they require training an additional model and/or the use of an additional component. zhang et al. (2022b) use reinforcement learning for demonstration selection of in-context learning; chang and jia (2023) train a scorer and estimator for demonstration selection; kumar and talukdar (2021) and xu et al. (2022) use a genetic algorithm; liu et al. (2022), lyu et al. (2023), and rubin et al. (2022) use retrieval from a corpus to select the prompts. on the other hand, probability-based prompt selection offers the advantage of prompt selection requiring only the output probabilities of the llm. while the prerequisite is a set of candidate prompts to select from, this data is relatively small in size and can be easily obtained from the research community (bach et al., 2022) or via machine generation (openai, 2023). one limitation of these methods, though, is that one cannot use them for closed-source llms that are only available via proprietary llm apis that do not provide output probability distributions. also, when the number of candidate prompts |t | and the size of the dataset used to select the prompt |x| is large, the calculation for prompt selection becomes computationally heavy; using a smaller set @xmath0. ## conclusion in this paper, we address the need for a comprehensive evaluation to compare the existing probability-based prompt selection methods, which have been proposed and evaluated under varying conditions and datasets. to achieve this, we introduce a unified evaluation setup to compare these methods, conduct a thorough evaluation, and develop a unified framework of the existing probability-based prompt selection methods. our analysis within this unified framework has provided insights into the relationship among existing methods, enabling the development of several combinational variants that improve performance. furthermore, our research on probability calibration has revealed the limitations of existing calibration methods and led to the proposal of an enhanced calibration method, calibration by marginalization (cbm). cbm not only significantly improves prompt selection performance but also demonstrates robust answer selection enhancement across multiple datasets. we hope that our unified setup provides a foundation for fair evaluation between various prompt selection methods and that our findings yield deeper insights into probability-based prompt selection.