Strength in Numbers: Estimating Confidence of Large Language Models by Prompt Agreement (2023)

Abstract: Large language models have achieved impressive few-shot performance on a wide variety of tasks. However, in many settings, users require confidence estimates for model predictions. While traditional classifiers produce scores for each label, language models instead produce scores for the generation which may not be well calibrated. We compare generations across diverse prompts and show that these can be used to create confidence scores. By utilizing more prompts we can get more precise confidence estimates and use response diversity as a proxy for confidence. We evaluate this approach across ten multiple-choice question-answering datasets using three models: T0, FLAN-T5, and GPT-3. In addition to analyzing multiple human written prompts, we automatically generate more prompts using a language model in order to produce finer-grained confidence estimates. Our method produces more calibrated confidence estimates compared to the log probability of the answer to a single prompt. These improvements could benefit users who rely on prediction confidence for integration into a larger system or in decision-making processes.

## introduction the modern framing of language modeling problems now includes the ability to perform numerous tasks previously handled by specialized supervised discriminative systems. for example, binary and multi-class classification tasks can be framed as text generation, where a large language model (llm) is given the input and the possible labels, and it generates the best label. more broadly, many reading comprehension, reasoning, and question-answering (qa) tasks can be framed in this multiple-choice style. an advantage to framing tasks in this manner is the ability to perform few-shot learning via in-context learning, in which a task can be performed based on only a handful of examples @xcite @xcite @xcite . instead of collecting a large dataset and investing time in training a new model, a user could utilize an existing llm for a new task by labeling a few examples and crafting a prompt: the input template which instructs the model to perform the given task. one challenge to utilizing llms in this manner is producing well-calibrated confidence scores for model predictions. a calibrated confidence score aids in the interpretation of model predictions @xcite and may be crucial if models become integrated into high-risk domains like healthcare and finance @xcite . a model is considered well calibrated if its prediction probabilities are aligned with the actual probability of its predictions being correct @xcite . if a model says an answer has 90% confidence, then we should expect it to be correct 90% of the time. formally, the probability that the predicted label ŷ is equal to the correct label y for input x should be equal to the model's predicted confidence @xcite . for supervised discriminative systems, confidence scores emerge from output probabilities or normalizing model scores to be between 0 and 1. for linear models, posterior probability serves as a reasonable confidence score because as the amount of evidence that supports prediction y increases, confidence also increases @xcite . however, prior work shows that these probabilities are not well calibrated for non-linear models @xcite . it is less clear how we can obtain confidence scores from llms. one approach is to use the (log) probability of the generation. however, these scores correspond to the likelihood of a text sequence given some context, as opposed to the actual probability of the label. for example, the model may assign probability mass to alternate generations that reflect the same answer (e.g. "answer a" vs. "the answer is a"). other creative approaches include asking the model to generate statements of confidence (e.g. "90% confidence in the label"), but it is unclear how to calibrate these open-ended statements @xcite . model self-consistency can be used to identify the most confident model output, but it is unclear how to produce a meaningful score @xcite . instead, we turn to another trend in llms: diverse prompts. @xcite showed that by writing variations of prompts for a range of tasks, they produced models better able to generalize to new domains. @xcite found that training on a diverse set of tasks improved model performance. we consider whether measuring the stability of an answer across a diverse set of prompts can be used to estimate model confidence. we propose to measure llm answer confidence by prompt agreement, whether the response of a model remains consistent across multiple prompts for a given instance. we prompt an llm with multiple different prompts that instruct the model to perform the same task for a single input instance and measure the agreement of the model responses across these prompts. we consider two approaches, represented in figure 1 . first, we measure the log probability of the response across multiple prompts that agree on the answer. second, we measure the diversity in answers across different prompts in the model output, concluding that answers which appear in more responses have relatively higher confidence. we compare these methods to the log probability of the answer produced in response to the official task prompt. we find that across a range of datasets and models, our methods consistently provide more accurate estimates of confidence. our contributions are as follows: ## related work we present the relevant background concepts of incontext learning and prompt sensitivity, and then outline approaches to confidence estimation. ## estimating confidence through multiple prompts we propose estimating model confidence through multiple prompts based on prompt agreement, i.e., the consistency among a model's generations in response to a set of diversely worded prompts. we prompt the model multiple times using different prompts, each of which asks the model to respond to a given question-answer (qa) input. intuitively, the more often that different prompts favor the same generation, the greater confidence the model has in that generation. for example, suppose that for a given question queried across ten prompts, the model always replies eggplant. for a second question queried with the same prompts, the model answers potato (5 times) and eggplant, cucumber, squash, carrot and kale. we would say the model is more confident in its answer to the first question. we score confidence via prompt agreement in two ways: (1) log probabilities across multiple prompts and (2) answer agreement across multiple prompts. we compare these to a baseline of the log probability of the response to a single prompt. ## models our confidence estimation methods are compatible with multiple language models. we evaluate our methods on three popular models, chosen because of their strong few-shot task performance, and focus on the largest models in each model "family" because they are the highest performing. for t0++ and flan-t5, we use the hugging face implementations locally. 2 t0++ is an 11b parameter t5-based model that was trained with a multitask mixture and multiple prompts on 55 datasets to improve zero-shot task generalization @xcite . flan-t5-xxl is an 11b parameter t5 model @xcite fine-tuned on 1.8k instruction oriented tasks @xcite . task finetuning (flan) produces state-of-the-art results on few-shot performance across several benchmarks. gpt-3 is a 176b parameter gpt-style model trained with a causal language modeling objective (brown et al., 2020). we use text-davinci-002, an instruction-tuned version of gpt-3 @xcite , through the openai api. due to the restrictions in obtaining all token logits in a single api call, we generate a model response and match it to the closest answer choice for cost efficiency. see appendix a for details. for each prompt and for each qa instance, we need to obtain (1) the model's selected answer from the multiple-choice list and (2) the log probability of the selected answer. to obtain the best answer we use rank scoring, which evaluates the model log probability for generating each answer from the multiple-choice list and selects the best option @xcite . for t0++ and flan-t5-xxl we use sanh et al. (2021)'s publicly available evaluation code, 3 modifying it to return log probabilities of the answers. we run these models on a compute instance with 4 a100 40gb gpus, with a per-device batch size of 8 for all datasets except dream (batch size of 1). finally, we omit results for automatically generated prompts for gpt-3 due to the high financial cost of using the api for so many prompts. we include these results for the other methods. ## data we evaluate our method across ten multiple-choice question-answering datasets. for each dataset, we have the official task prompt and a source of diverse prompts for the same task. within a dataset, each instance contains contextual information, a series of multiple-choice answers, and annotations indicating the correct answer. we use the following multiple-choice qa datasets from the t0 training mixture @xcite : cos-e v1.11 (rajani et al., 2019), cosmos qa @xcite , dream @xcite , qasc @xcite , quail @xcite , quarel @xcite , quartz @xcite , sciq @xcite , social iqa @xcite , and wiqa @xcite . we exclude wikihop @xcite due to the extra computational resources needed for this dataset. we use only the validation splits. ## prompts we pair these datasets with three sources of prompts: the official task prompt and two sources for diverse set prompts for each task. first, we use the official task prompt as defined in the original paper for each dataset. second, we use the diverse human-authored prompts provided by @xcite . each prompt is a template that contains text strings and placeholders to insert the question and answer choices (see figure 2 ). we only use the t0 prompts that correspond to the original task intended by the dataset's authors. we refer to these as the multiple human prompts. we apply these prompts to the qa data using the promptsource library @xcite and evaluation code for t0. 3 third, we create a larger set of prompts through automated prompt generation. while having multiple prompts leads to better confidence scores, not every task has multiple human-authored prompts available. furthermore, if multiple prompts are helpful, perhaps a larger set would provide more fine-grained confidence estimates. automatically generating prompts addresses both of these cases. many methods have been proposed for automatically generating llm prompts. most prompt generation methods assume either a single prompt for a task @xcite @xcite or a unique prompt for each input @xcite . instead, we seek to generate multiple prompts for each task. we draw inspiration from the iterative prompt generation process of @xcite , which generates paraphrases of a prompt by asking a llm to paraphrase instructions with different prompt generation prompts (pgp). for example, by providing an llm the pgp "generate a variation of the following instruction while keeping the semantic meaning," we can obtain prompt variations. we use a total of 15 pgps (listed in we use this method with gpt-3 text-davinci-002 to generate a set of automatically generated prompts (agps) based on 31 instruction statements extracted from the t0 prompts (listed in ## evaluation does measuring confidence across multiple prompts yield better calibrated confidence scores? a common approach to measuring calibration is expected calibration error (ece) @xcite , which buckets the prediction probabilities and measures the empirical accuracy of each bucket with its average estimated probability (confidence). 4 the discrepancy between these terms is the calibration gap; lower gaps indicate better calibration. ece ranges from 0 (perfect calibration) to 1 (lowest calibration). we utilize this method to compare log probabilities obtained from a single prompt to those from multiple prompts. for each dataset, we use 10 evenly-spaced bins and set the min and max of the bins according to the minimum and maximum average log probability in the dataset. we measure agreement across prompts using rand index, which does not give normalized scores that can be interpreted as probabilities. we could convert these scores into probability confidence scores in two ways. 1) measure the empirical accuracy of different ranges of rand index on a held-out validation set, then assign confidence scores based on those accuracies. the drawback to this approach is it requires a separate held-out set for calibration, which may be an unrealistic assumption, especially in few-shot settings. 2) normalize the empirical rand index scores to form a probability distribution. we experimented with this approach but found that how we bucketed and normalized the scores heavily influenced ece results, which produced an unfair evaluation setting. instead, we view rand index scores as a relative confidence score between instances, where a higher score means "more confident." we propose an evaluation metric that considers relative confidence of answers between instances. we rank instances in a dataset according to their confidence scores (log probability or rand index), with the highest scoring instance (e.g., largest log probability or rand index) at the top of the list. we evaluate each confidence estimation method on how well it ranks correct predictions higher than incorrect ones. most evaluation metrics for ranking are geared towards an information retrieval setting where the number of items in the list can vary, different items can be included by each model, only a few items are "relevant", or we have close to a total ordering over the ranked items. our ranked lists differ significantly from these settings. therefore, we choose a simple, intuitive ranking evaluation: swapped pairs, based on the ranking loss function from díez @xcite ; @xcite . a list is scored based on the number of item pairs that need to be swapped to create a correct ordering. this penalizes methods that have higher confidence in predictions that were incorrect over correct predictions. swapped pairs is not normalized and grows with the number of items in a ranked list (from 0, i.e., perfect rank ordering, to n * (n-1) 2 , i.e., worst rank ordering, where n is the number of items to be ranked). we report macro-averaged results by dividing the total swapped pairs by dataset size, after filtering out any invalid predictions. 5 8 results multiple prompts provide a more calibrated confidence estimate than a single prompt. measuring confidence using prompt agreement with human-authored prompts also improves over using a single prompt as measured by swapped pairs. there is not a clear winner between the log probability and agreement methods, as each obtains the most calibrated scores for some models. however, both ways of using multiple human-authored responses improve over a single prompt. automatically generated prompts show mixed results. sometimes automatically generated prompts improve over a single prompt (ece on flan-t5-xxl), and sometimes they do not. we suspect that this may be related to the quality of the prompts. poorly written prompts that obtain worse accuracy on the task give worse confidence scores. to test this hypothesis, for each dataset we select the top 10 prompts with the highest accuracy on the validation set. we compare the confidence scores from using these 10 prompts with the scores from using all agps. however, this filtering still does not yield consistent improvements on ece or swapped pairs. there may be other factors that prevent automatically generated prompts from producing better confidence scores. for example, they may have insufficient diversity or may be worse in some other manner. in contrast, we know that the human-authored prompts were carefully written by people who have experience prompting language models. despite the poor performance of agps, they still show improved performance over a single prompt, indicating that agps could serve as a substitute for human-authored prompts if human-authored prompts are not available. ## conclusion our experiments with t0++, flan-t5-xxl, and gpt-3 suggest that prompt agreement provides a more calibrated confidence estimate than the typical approach of log probability from a single prompt. we find mixed results in scaling up the number of prompts using automatically generated prompts. experimenting with additional prompt generation methods may enable the automatically generated prompt approach to produce even better calibrated confidence scores. we leave this to future work.