Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness (2024)

Abstract: We introduce BSDetector, a method for detecting bad and speculative answers from a pretrained Large Language Model by estimating a numeric confidence score for any output it generated. Our uncertainty quantification technique works for any LLM accessible only via a black-box API, whose training data remains unknown. By expending a bit of extra computation, users of any LLM API can now get the same response as they would ordinarily, as well as a confidence estimate that cautions when not to trust this response. Experiments on both closed and open-form Question-Answer benchmarks reveal that BSDetector more accurately identifies incorrect LLM responses than alternative uncertainty estimation procedures (for both GPT-3 and ChatGPT). By sampling multiple responses from the LLM and considering the one with the highest confidence score, we can additionally obtain more accurate responses from the same LLM, without extra training steps. In applications involving automated evaluation with LLMs, accounting for our confidence scores leads to more reliable evaluation in both human-in-the-loop and fully-automated settings (across both GPT 3.5 and 4).

## introduction while the promise of large language models (llms) and agents (powered by llms) has become evident, their usage in high-value applications remains limited by their unreliability. accessed via black-box apis (via providers like ope-nai/anthropic), today's best llms have been trained to produce convincing-looking responses and thus often appear overconfident @xcite . for many input prompts encountered in the wild, the model cannot be certain about the desired response (perhaps because the prompt is vague or is related to a specific fact/event absent from the training dataset), yet these models output plausiblesounding yet wildly incorrect answers in such scenarios. this hallucination problem has also plagued traditional supervised learning systems, where it is traditionally addressed via uncertainty estimation to know when one can trust a model's prediction @xcite @xcite @xcite @xcite . in traditional supervised learning, one has access to the training data of the model and its probabilistic estimates, as well as being able to modify the training procedure to improve model calibration @xcite . other traditional uncertainty estimation procedures require the existence of a validation set that can be used for calibration @xcite . none of this is available for today's best llms, which may be given any imaginable prompt rather than (input, output) pairs stemming from a limited distribution. thus approaches to uncertainty estimation for black-box llms must wrap the inference procedure. our proposed llm's uncertainty quantification technique, bsdetector, calls the llm api multiple times with varying prompts and sampling temperature values (see figure 1 ). we expend extra computation in order to quantify how trustworthy the original llm response is, a worthwhile tradeoff for high-stakes applications. our method is conceptually straightforward, generally applicable across llm providers (as well as agent frameworks @xcite or any stochastic @xmath0, and produces confidence scores whose values are reliably lower for responses from the llm that are more likely bad. bsdetector confidence scores allow llms to be more safely used in high-stakes applications, since we can know which llm outputs are not to be trusted. depending on the application, we can adaptively ask a human for an alternative response when the confidence score is low, automatically route the prompt to an alternative llm provider, or simply respond "i don't know" when a confident response cannot be generated. our experiments reveal that for question-answering applications, we can automatically generate more accurate answers by sampling multiple responses from the same llm and selecting the response whose bs-detector confidence estimate is the highest. this paper primarily focuses on question-answering applications, but our same uncertainty estimates can also be applied to estimate how confident the llm is in its response to a more general prompt. intuitively, we'd like to see a low confidence score when the llm outputs: a factually incorrect response to a question, a inaccurate summary requested for a document, or a generated article/message that semantically differs from the intention of the original request. ensuring this is challenging without control over llm training, but we can hope that in each of these three scenarios where the model generated a bad response, a well-trained llm was also likely to output alternative responses (which more closely reflect the desired response). bsdetector is based on this intuition, and is observed to produce effective uncertainty estimates with today's top llms from openai across prompts from closed and open domain benchmark datasets. ## related work for estimating the confidence levels tied to responses output by large language models, @xcite introduces semantic entropy, incorporating linguistic invariances created by shared meanings. however their approach requires access to token-level probabilities from the llm, which is often not accessible with today's blackbox apis. @xcite prompts the models to self-evaluate their answers and directly ask the llm to produce the likelihood p (answer is true) -also fine-tuning the model to output better values for its stated likelihood. relatedly, @xcite prompts llms to generate both an answer and a level of confidence. manakul et al. (2023) proposes a sampling-based approach to detect hallucinated facts. all of these aforementioned approaches train additional models via supervised learning, unlike bsdetector which does not employ any additional training. more recently, tian et al. (2023) conducts evaluations of computationally feasible methods to extract confidence scores from the probabilities output by llms trained via reinforcement learning with human feedback. @xcite differentiates between uncertainty and confidence estimation for llms (under their terms, our work is focused on the latter, but without requiring access to the auto-regressive token probability estimates their method is based on). the works of @xcite and @xcite only study limited tasks, and it remains unclear whether their conclusions still hold in the context of reasoning or arithmetic. here we demonstrate that our method produces effective uncertainty estimates across multiple domains involving reasoning, arithmetic, and knowledge of facts. ## bsdetector uncertainty estimation when posing a question to llms, we aim to to estimate how confident we should be that a particular llm answer is correct (or simply "good" for more general llm responses). specifically, for input question x, we want to not only obtain an answer y from the llm, but also an associated confidence score for this answer c(x, y). our confidence assessment derives from two factors: observed consistency and self-reflection certainty, which respectively are extrinsic and intrinsic evaluations of llm confidence. since a welltrained llm should consider multiple different answers when asked an under-specified question or about something not contained in its training data, observed consistency extrinsically measures whether the llm finds multiple contradictory answers likely to be good responses. since effective llms can reasonably evaluate text from arbitrary agents, self-reflection certainty directly asks the llm to intrinsically reflect on whether its own previously-generated answer seems correct and how confident it is about this. ## application: generating more reliable answers from any llm one straightforward application of our bsdetec-tor uncertainty estimation is to apply it to (each of) multiple candidate answers produced from the same llm: {y ′ 1 , y ′ 2 , ..., y ′ k }. this assessment allows is to determine which candidate llm answer y ′ i appears most trustworthy, and return that one instead of always returning y (see figure 2 ). specifically, we use the same prompt to ask the llm to produce several responses via temperature sampling. for each candidate answer, we reuse the same set of previously-described llm outputs {y 1 , y 2 , ..., y k } to compute an observedconsistency score. following the standard bsde-tector procedure, we prompt the llm to assign a self-reflection certainty to each candidate response. finally we select the answer with highest bsde-tector confidence score. an alternate answer y ′ i ̸ = y can be deemed most trustworthy via this procedure only if: the llm was able to identify fewer likely answers that contradict y ′ i and was more certain about the correctness of y ′ i during the intrinsic self-reflection assessment. ## application: more reliable llm-based (automated) evaluation in open-domain tasks, it is challenging to evaluate the correctness/quality of answers (irrespective of whether these answers were generated by a llm or human). often one resorts to automated evaluation using models like gpt-3.5-turbo or gpt-4 to assess the correctness of answers @xcite @xcite @xcite . recent instruction fine-tuning techniques such as alpaca @xcite and wizardlm @xcite also utilize gpt-4 for automated evaluation of generated answers. even when they are based on advanced llms like gpt-4, there remain questions about the reliability of these llm-based evaluations. here we outline two ways to boost the reliability of llm-based evaluation: human-in-the-loop and fully automated. both start by computing bsdetector confidence scores for each llm-evaluation (these scores estimate not the trustworthiness of the generator of the answers, but rather the evaluator of their correctness). let a denote the subset of 6 experiments 6.1 calibration of uncertainty estimates datasets. our experiments consider numerous question-answering benchmarks listed below. for each example in each benchmark dataset, the true answer is known enabling us to precisely assess the accuracy of llm responses. we study performance in: gsm8k @xcite and svamp @xcite , datasets composed of grade school math word problems, commonsense question answering (csqa) @xcite , a dataset requiring some level of reasoning, and triviaqa @xcite , an open-form trivia question dataset that gauges models' factual knowledge. because triviaqa is open-domain, the correct answers provided do not entail all valid solutions, so we also manually validated the accuracy of llm-generated responses. baseline methods. our study also evaluates the following baseline uncertainty estimation methods: likelihood based uncertainty calculates the joint log-probability of a sequence from the autoregressive estimator and normalizes it by the sequence length @xcite . while it represents the typical way to estimate aleatoric uncertainty in traditional supervised learning and structured prediction @xcite , this approach can only can be applied to text-davinci-003, since the gpt-3.5 turbo api does not provide access to token-level probabilities from the model. self-reflection certainty @xcite and bsdetector are introduced in fig 1a . temperature sampling @xcite is equivalent to bsdetector without: cot prompting, self-reflection certainty, and the indicator function term inside of the text-similarity metric. results. ## comparison with related work and further impact of our work it is very challenge to benchmark some llm baselines, this difficulty often arises due to the extensive use of meticulously crafted prompt engineering in some papers. our goal is not necessarily to out-perform some baselines, as prompt engineering can significantly impact results. instead, we strive to minimize task-specific, ad-hoc processes in favor of creating simple, yet effective methods. in our paper, we concentrate exclusively on question-answer scenarios because their performance is easily measurable. however, in practice, bsdetec-tor is applied across a wider range of applications. for instance, we applied our method to assist attorneys in drafting documents ( ## discussion this paper presents bsdetector, a method designed to identify unreliable or speculative answers from llms by computing a confidence score for its generated outputs. due to its simplicity and generality, we expect bsdetector uncertainty estimation to find many applications across diverse domains/tasks, beyond the studies in this paper on.