Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words? (2024)

Abstract: We posit that large language models (LLMs) should be capable of expressing their intrinsic uncertainty in natural language. For example, if the LLM is equally likely to output two contradicting answers to the same question, then its generated response should reflect this uncertainty by hedging its answer (e.g., “I’m not sure, but I think...”). We formalize faithful response uncertainty based on the gap between the model’s intrinsic confidence in the assertions it makes and the decisiveness by which they are conveyed. This example-level metric reliably indicates whether the model reflects its uncertainty, as it penalizes both excessive and insufficient hedging. We evaluate a variety of aligned LLMs at faithfully conveying uncertainty on several knowledge-intensive question answering tasks. Our results provide strong evidence that modern LLMs are poor at faithfully conveying their uncertainty, and that better alignment is necessary to improve their trustworthiness.

## introduction despite their unprecedented capabilities, large language models (llms) often output erroneous information @xcite @xcite . moreover, llms typically communicate such inaccurate information in a fluent, decisive, and persuasive manner, which may lead users to overly rely on their false output @xcite . we argue that a possible pathway for improving llm trustworthiness is to have the model communicate its uncertainty in words, as part of its generated response @xcite . expressing uncertainty in natural language has several benefits over using numerical estimates. first, language provides a rich space, which can be useful to convey the source of the model's uncertainty. second, it is generally perceived as more intuitive to humans @xcite 1958. i think he was born on @xcite , but i'm not sure. december 2, 1981. possibly in 1981. ## faithful response uncertainty our goal is to evaluate whether models can express uncertainty in words to faithfully reflect their intrinsic uncertainty. to this end, we first propose to consider the decisiveness with which assertions in a response are expressed. given a query q and a response r generated by a model m , we view r as a sequence of assertions a(r) = {a 1 , ..., a n }, each expressed with some level of decisiveness that is derived from possible hedging expressions associated with it. 1 for example, given the query "tell me about barack obama", the response "barack obama is an american politician. i think he was born in 1961, but i'm not sure." contains two assertions: a 1 = barack obama is an american politician, and a 2 = barack obama was born in 1961. while a 1 is conveyed decisively in the response, a 2 is less decisive due to the hedging expressions "i think" and "i'm not sure". we consider a response r as faithful to m if for every assertion @xmath0, the decisiveness in which a is conveyed matches m 's intrinsic confidence in a: definition 1 (faithful response uncertainty) for a query q and a response r generated by a model m , the faithfulness of r with respect to m 's intrinsic confidence is given by: note that faithfulness (shorthand f) is in [0, 1], where a maximal value of 1 is obtained when every assertion's decisiveness matches the model's intrinsic confidence. lower faithfulness values are obtained in cases of unnecessary hedging, that is, expressing uncertainty in assertions that the model is certain about, or lack of hedging, i.e., not expressing uncertainty in assertions the model is not confident about. see examples in fig. 1 . ## measuring decisiveness & uncertainty we now propose an implementation to the faithfulness score, focusing on the setting of short-form question answering (qa), where q is a factual question (e.g., "when was barack obama born?") and r is typically a short answer with a single (possibly hedged) assertion (e.g., "august 1961"). quantifying decisiveness prior work quantified the decisiveness of an assertion as a binary notion, based on whether the assertion is accompanied by a hedging expression or not @xcite . however, this captures only little of the expressivity through which hedging expressions can convey uncertainty. recognizing that decisiveness is subjective in nature, we draw inspiration from definitions of veridicality @xcite and propose the notion of perceived decisiveness, which aims to be relativized to particular agents or perspectives. formally, we define the perceived decisiveness of an assertion @xmath1: quantifying uncertainty following previous work @xcite @xcite , we quantify certainty via consistency. concretely, for a query q (e.g., "when was barack obama born?") we quantify the uncertainty of a generated assertion a (e.g., "barack obama was born in 1961") by examining the consistency between this assertion and re-sampled answers to q: if the generated answers agree with a (e.g., "1961", "i think he was born in 1961", or "august 4, 1961."), then we say m is confident in a. conversely, assertions that contradict a (e.g., "1962" or "probably 1955") indicate that m 's confidence in a is lower. 2 formally, given a question q and a generated response r consisting of a single assertion a, let {r 1 , . . . , r k } be the set of sampled responses and {a 1 , . . . , a k } the set of corresponding assertions (i.e., a(r i ) = {a i }). we quantify the confidence of m in a as the fraction of sampled assertions that contradict a: implementation details we implement the above scores (eq. 1, eq. 2) by prompting a "judge" llm. for a given query q and a generated response r, we first extract the assertion a in r and its decisiveness score using a few-shot prompt p d (see tab. 5 in §d). next, to quantify the model's intrinsic confidence, we sample @xmath2. then, we use another few-shot prompt p c (see tab. 6 in §d) to check for every extracted assertion whether it contradicts a. in our experiments, we use gemini ultra as the judge. correlation with human judgement we evaluate the quality of our llm-based scores, showing that they correlate well with human judgment. for decisiveness (eq. 1), we randomly sample 100 model answers generated in our experiments ( §4) and rewrite each answer to include a hedging expression (e.g., "highly likely"). then, we score answers with our decisiveness prompt p d . fig. 2 shows for each hedging expression the mean decisiveness score versus the distribution of perceived probabilities humans assigned to it (using survey data from fagen-ulmschneider (2023)). overall, the llm scores agree with the human evaluations. for confidence (eq. 2), we compare the confidence scores for 100 randomly selected examples, when calculated with our prompt p c versus when using labels written by the authors. we observe a high correlation of 0.97 between the two scores. ## experimental setting we evaluate whether llms faithfully reflect their uncertainty when answering questions. data we use knowledge-intensive qa datasets: as we focus on model uncertainty, foot_2 we exclude ambiguous questions for which uncertainty can rise due to data uncertainty (see details in §a). ## results in tab. 1 we report our faithfulness metric (cmfg) for all model-method-dataset combinations. without special instructions, models generate decisive answers, even for uncertain answers considering the vanilla baseline, all models perform poorly in terms of faithfulness with cmfg close to 0.5 (tab. 1, top row). this happens because all the models we tested did not generate any expressions of uncertainty, despite having significant intrinsic uncertainty in some cases. see fig. 3 . state-of-the-art models cannot be easily steered towards faithfully expressing uncertainty via prompting. for the non-vanilla baselines, there (2) even the best models have some significant uncertainty (e.g. on the challenging popqa benchmark, the high confidence is 0.8). figure 4 : weak correlation between decisiveness and confidence: we plot decisiveness (y-axis) vs confidence (x-axis) for two of the best performing (model, method, dataset) combinations (see ## conclusion we formalize the desiderata that a language model's generated response should reflect its intrinsic uncertainty in natural language. we instantiate this with a generic evaluation framework that quantifies the mismatch between the linguistic decisiveness of the assertions made in the response and the intrinsic uncertainty the model has in these assertions. we quantify intrinsic uncertainty by considering the consistency of model answers across multiple samples, but our framework is generic and can be extended to other methods for quantifying