B ayesian Prompt Ensembles: Model Uncertainty Estimation for Black-Box Large Language Models (2024)

Abstract: An important requirement for the reliable deployment of pre-trained large language models (LLMs) is the well-calibrated quantification of the uncertainty in their outputs. While the likelihood of predicting the next token is a practical surrogate of the data uncertainty learned during training, model uncertainty is challenging to estimate, i.e., due to lack of knowledge acquired during training. Prior efforts to quantify uncertainty of neural networks require specific architectures or (re-)training strategies, which are impractical to apply to LLMs with several billion parameters, or for black-box models where the architecture and parameters are not available. In this paper, we propose Bayesian Prompts Ensembles (BayesPE), a novel approach to effectively obtain well-calibrated uncertainty for the output of pre-trained LLMs. BayesPE computes output probabilities through a weighted ensemble of different, but semantically equivalent, task instruction prompts. The relative weights of the different prompts in the ensemble are estimated through approximate Bayesian variational inference over a small labeled validation set. We demonstrate that BayesPE approximates a Bayesian input layer for the LLM, providing a lower bound on the expected model error. In our extensive experiments, we show that BayesPE achieves significantly superior uncertainty calibration compared to several baselines over a range of natural language classification tasks, both in zero- and few-shot settings.

## introduction generative pre-trained large language models (llms) have proven to be very effective at performing a broad variety of natural language processing (nlp) tasks in zero-and few-shot settings @xcite @xcite . these capabilities make them attractive alternatives to task-specific fine-tuned models, as they do not require large annotated data sets or gradient updates, but simply take as input relevant text instructions (i.e., prompts) to perform the inference of interest @xcite @xcite . however, safely deploying llms requires wellcalibrated estimates of the uncertainty in their outputs @xcite @xcite . this is particularly challenging as llms are normally not trained to capture their own uncertainty and, due to their size and complexity, it is difficult and expensive to fine-tune or adapt them @xcite @xcite . in addition, many state-of-the-art llms are proprietary and available only via apis with their weights and architecture hidden from end users, leaving black-box integration as the only option @xcite . this precludes the application of common model uncertainty estimation methods for deep learning, i.e., drop-out or deep ensembles @xcite @xcite . to overcome these limitations, we propose bayesian prompts ensembles (bayespe), a simple yet effective approach for estimating the uncertainty of llms in black-box scenarios (i.e. without access to model weights). similarly to @xcite and @xcite , we ensemble the output probabilities obtained by running the llm classifier with different variations of the prompt task instructions. however, instead of simply combining all prompts with equal contribution, we learn their relative importance through approximate variational inference with a small validation set. figure 1 schematically illustrates our approach. bayespe adapts the use of different prompts to the task and llm of choice, while obtaining well-calibrated estimates of the output uncertainty. we further show that bayespe approximates a bayesian input layer to the llm and provides a lower bound to the reducible error of the model. in our extensive empirical evaluation, we test bayespe with several llms and a range of text classification tasks, both in zero-and fewshot scenarios. in each case, we measure several uncertainty estimation and calibration metrics and compare to existing competitive baselines. model uncertainty in deep learning developing deep neural networks (dnns) that "know when they don't know" is a long standing challenge in deep learning @xcite . unlike other machine learning models, such as gaussian processes @xcite @xcite and markov chain monte carlo samplers @xcite , dnns are not capable to self-assess output uncertainty originating from model misspecification, i.e., model or epistemic uncertainty, often resulting in wrong inferences with high confidence for out of distribution data @xcite . conversely, bayesian neural networks (bnns) provide a principled way to capture this uncertainty by assigning distributions of values to their weights, instead of point estimates @xcite @xcite . bnns then estimate epistemic output uncertainty by sampling the weights' distributions at inference time. while bnns are arguably the most principled way to model uncertainty, they are difficult and often prohibitively expensive to train in practice @xcite @xcite . this motivated several approximations to bnns to efficiently model epistemic uncertainty. most success-ful approaches include drop-out, where variation in the weights is approximated by masking random hidden units @xcite , and deep ensembles, where uncertainty is estimated with multiple models trained on the same data with different random seeds @xcite . in this work, we aim to approximate model uncertainty for black-box pre-trained llms. in this setting, uncertainty is intractable and we assume no access to the weights to modify or train the model. uncertainty quantification for llms one way to estimate uncertainty for black-box llms is to use the probabilities of the generated tokens @xcite @xcite . using relative tokens probabilities is proven to provide good calibration for several tasks @xcite . recent work also models the uncertainty of full sentences taking into account semantic equivalence. malinin and gales (2020) use input-output mutual information to capture uncertainty. @xcite compute distributions and uncertainty in a representation space specifically trained such that distances capture semantic similarity. @xcite uses the llm itself to evaluate similarity among multiple generations. using the llm token probabilities is a reliable estimate of the uncertainty they learn from training data, i.e., data uncertainty. however, it does not necessarily capture model uncertainty, i.e., for inputs where the model is not competent @xcite . arguably, model uncertainty is more critical to safety, as it assesses when we can trust the predictions of the llm @xcite . analogously to approximate bayesian approaches for deep learning, several studies propose sampling weights with dropout @xcite . other approaches build and train specific components instead, designing bayesian transformer blocks @xcite or adding an uncertainty component to existing pre-trained llms and fine-tuning the whole architecture @xcite . while effective, these approaches need to have access to the model weights and are computationally expensive to apply at scale. a recently proposed approach for black-box scenarios is that of prompting llms themselves to generate estimates of their own confidence @xcite @xcite . however, it is not trivial to construct prompts that result in accurate confidence assessments @xcite . prompt ensembles recent works have explored aggregating prompts to improve predictive performance of llm prompting @xcite . similarly to our own work, @xcite assign different weights to different prompts in the ensemble and optimize them with a validation set. however, they train specific verbalizers for each prompt and their objective is based on classification accuracy maximization. in our own work, we perform direct prompting instead and prompts weights are optimized with a bayesian objective, balancing data likelihood and uniform exploitation of the prompts for improved calibration. prompts ensembles have also been explored to improve calibration, studying different strategies to generate diverse prompts for this purpose @xcite . however, these works aggregate all prompts in the ensemble with equal importance. as llms are highly sensitive to prompt engineering @xcite , this approach runs the risk of including low performing prompts in the ensemble and degrade overall performance. conversely, we hypothesize that by learning the importance of different prompts with a bayesian approach, we can strike a balance between prompt variation and data likelihood maximization, in analogy with the inference of bnn weight distributions given a prior and data. ## bayesian prompt ensembling problem description we consider the problem of text classification with a black-box decoder-only llm via prompting. in a zero-shot setting, the text input (prompt) to the llm can be separated into two parts: (i) task instructions a, e.g., "is the sentiment of the following review positive or negative?"; and (ii) test input x, e.g., a movie review in a test set. the prompt a is constant to all inputs for a given task, while test inputs x change, e.g., different reviews to be classified as positive or negative. the class probabilities p(y|x, a) for the corresponding label y are inferred by passing the instruction-test data point pair {a, x} as input to the llm, e.g., "is the sentiment of the following review positive or negative? {review}", to obtain the probabilities for the next token or sequence that correspond to the particular classes in the vocabulary, e.g., 'positive' and 'negative'. in a few-shot setting, the task instructions can be accompanied with demonstrations (i.e. a small set of labeled data points). we also assume that we have access to n semantically equivalent task instruction prompts a @xmath0, and a small validation data set d of m data points x @xmath1. our main aim is to optimally exploit the llm through prompting p(y|x, a), the available labeled validation data d and the given batch of prompt instructions a to obtain well-calibrated classification probabilities p(y|x, d). by well-calibrated, we mean that the llm assigns, on average, the same confidence to its own prediction as the hidden ground-truth, i.e., e p(y|x,d) log p(y|x, d) ≃ log p(y * |x, d) for new test input data {x, y * }. this results in a model which produces high confidence when it generate the correct class token and low confidence when it is likely to generate the incorrect class token. experimentally, we measure this property with a variety of established calibration metrics. in this paper, we do not aim to optimize the definition or automatic generation of the prompts a. we assume these to be fixed and equally relevant a-priori. we focus on the problem of how to optimally employ them for well-calibrated inference. we study performance with different existing prompt generation strategies in appendix c.3. ## experimental setup data: we evaluate bayespe on a total of ten tasks. three sentiment analysis tasks: amazon reviews (he and mcauley, 2016), @xcite and @xcite . three topic modeling tasks: dbpedia 14, yahoo answers @xcite and trec @xcite . two semantic relation tasks: mrpc @xcite and @xcite . finally, two spam detection tasks: sms @xcite and youtube @xcite . these data sets cover a variety of common nlp tasks, number of classes and text input lengths (details in appendix b.1). baselines: we compare bayespe to the following baselines: 1 prompts and validation data: we consider a fixed set of n initially provided prompts a i for each task. we do not aim to investigate prompt definition or engineering and, unless stated otherwise, we simply draw these at random from a pool of prompts generated with gpt-3.5-turbo @xcite with no manual editing. the prompts are used to compute ensemble, best and bayespe, while standard is computed by drawing just one prompt at random. for few-shot experiments, the prompts include five labeled examples from a different pool of validation data. these are drawn at random for each prompt in the set (details in appendix b.3). in each task, unless stated otherwise, we use @xmath2, 50 validation examples to select the prompt for best and train the bayespe weights (details in appendix b.4), and 200 test examples to compute metrics. the set of prompts and validation examples are drawn at random ten times in each of our experiments to obtain error bars and compute statistical significance of results. method auc (↑) ece (↓) mce (↓) ensemble -3.9% -20.0% -16.6% best +15.8% -26.3% -17.0% bayespe +23.7% -32.1% -23.3% ## results uncertainty calibration we evaluate performance of bayespe and baselines on all tasks with all llms in zero-and few-shot settings. in the trends described above are observed across all experiments. while inspecting all results in detail is difficult (included in appendix c.5), we present aggregate results, both in terms of average differences in calibration and statistical significance. ## conclusion we proposed bayesian prompt ensembles (bayespe), an effective method to approximately estimate the model uncertainty for classification 12236 with black-box llms. we showed that bayespe constitutes a discrete approximation to a bayesian input layer and that its uncertainty gives a lower bound on the model error, analogous to that of bnns. bayespe is significantly better or comparable to competitive baselines across the majority of experiments. in our analysis, we showed how bayespe improves uncertainty quantification over baselines as both the inference and labeled data budget increase.