On the Role of Unobserved Sequences on Sample-based Uncertainty Quantification for LLM s (2025)

Abstract: Quantifying uncertainty in large language models (LLMs) is important for safety-critical applications because it helps spot incorrect answers, known as hallucinations. One major trend of uncertainty quantification methods is based on estimating the entropy of the distribution of the LLM’s potential output sequences. This estimation is based on a set of output sequences and associated probabilities obtained by querying the LLM several times. In this paper, we advocate and experimentally and show that the probability of unobserved sequences plays a crucial role, and we recommend future research to integrate it to enhance such LLM uncertainty quantification methods.

## introduction the advent of large language models (llms) has revolutionized numerous fields by demonstrating remarkable capabilities across a diverse array of tasks. however, despite their impressive performance, these models often struggle with reliability issues, particularly due to factual inaccuracies in their outputs. in this context, quantifying their confidence and adjusting them for various tasks can reduce risks and enhance the quality of outputs. however, uncertainty quantification (uq) on llms remains challenging since the output possibilities for these models are substantially greater than those of discriminative models. as the generation length increases, the number of potential outcomes grows exponentially, making it unfeasible to evaluate all possible answers @xcite . we can distinguish two types of uncertainty in llms: aleatoric uncertainty, stemming from inherent randomness, and epistemic uncertainty, resulting from a lack of information @xcite . following previous work, we aim to quantify a measure of total uncertainty, i.e., aleatory and/or epistemic, as both types of uncertainty contribute to model errors. among the methods of uncertainty quantification for llms, we identify black-box methods, which assume access only to the generations, and white-box methods, which also utilize internal states of the llm or token-level probabilities. in this paper, we focus on the latter, utilizing tokenlevel probabilities. concretely, we study samplingbased estimation methods, that is, approaches that rely on information (e.g. probability) obtained from multiple answers generated by the llm, in order to quantify uncertainty. sample-based uncertainty quantification methods via entropy estimation, like predictive entropy (e) @xcite and the recently proposed semantic entropy (se) @xcite , have succeeded recently perhaps due to their simplicity, as they do not require any special training or architectural modifications. however, we note that these methods are themselves subject to epistemic uncertainty, as they rely on only a glimpse of the probability distribution of possible answers due to practical constraints. we highlight that methods like e and se, in particular, do not account for this epistemic uncertainty, as they only consider the estimated probability of sampled sequences, neglecting the remaining probability of possible but unobserved answers. recent work by abbasi yadkori et al. (2024) has moved in a similar direction and explored the concept of missing mass in uq. however, their approach directly compares the distributions of the generated answers against the ground truth. instead, here we present work focusing on modeling the probability of unobserved answers without the need for ground truth. concretely, our aim is to propose a framework that enables us to incorporate this probability into existing formulations for estimation based on entropy. we provide technical considerations for the calculation of such probability and evaluate the relevance of one such implementation by using it as a uq method. ## proposed approach let us denote by x the object about which we quantify uncertainty; in our case study, x refers to the input given to the llm which often consists on a question and potentially a prompt. we denote by (s, p) the probability space, where s is the set of all possible sequences, and p is the probability measure over s. the entropy for the random output sequence of the llm and the input x is defined as equation 1 shows, below, where p(s|x) is the probability of the sequence s conditioned on the input x. as it is not realistic to compute the probability of all answers in s, entropy-based uq methods for llms estimate e * (x) on a set of m sequences sampled from the model denoted s 1 , . . . , s m . let us denote a ⊂ s the set of unique sampled answers and note that |a| ≤ m because some identical answers might be sampled multiple times. each answer @xmath0. the probability of @xmath1, . . . t n ) is obtained by the product of conditional token probabilities via the language model, as follows. some works have considered adjusting the calculation of sequence probabilities to account for varying sequence lengths. this is due to the tendency for longer sequences to exhibit lower joint likelihoods. to address this, a length normalized probability, which we denote p ′ was proposed @xcite as follows. we now focus on the probability of sequences not observed in the set a of sequences provided by the llm for a given input x. this probability is given by we believe that the probability of unobserved sequences can capture some of the uncertainty associated with an input x. when uncertainty is low, the model's output probabilities tend to be higher, leading to a lower probability for the unobserved sequences. conversely, when uncertainty is high, the model's output probabilities are lower, resulting in a higher probability for the unobserved sequences. in case of maximum uncertainty, all sequences in s are equally likely, with each having a probability of 1 /|s|. as a result, p( ā|x) = 1 -m /|s| approaches 1, especially when the set of possible sequences is very large. conversely, in situations of minimal uncertainty, p( ā|x) = 0. in practice, we have two technical concerns related to the accurate calculation of probabilities for unobserved answers. firstly, to the best of our knowledge, it is not always clear whether the last token, specifically the end-of-sequence (eos) token, is considered in sequence probability calculations presented in equation 2. if sequences do not include the eos token, this raises concerns about the construction of the sample space, as two unfinished sequences are not mutually exclusive. let us introduce a small example to illustrate our discussion about the sequence probability calculation. example. for the question input @xmath2. peter's basilica and the sistine chapel?", let us assume we observed two output sequences such that @xmath3, "vatican city"} and consider the token conditional probabilities presented in figure 1 . if we do not include the end-of-sequence token, the probability value of 0.8 may be incorrectly interpreted as the probability of the sequence "vatican". in fact, this represents the probability that the sequence starts with "vatican", which also includes the possibility of the sequence being "vatican city". essentially, the events of the sequence beginning with "vatican" and "vatican city" are not mutually exclusive. in addition to this issue, we also note that sequence length normalization techniques as shown in equation 3, and often used approaches like se, can distort probabilities, potentially leading to the sum of output probabilities differing from 1. due to the issues discussed above, we highlight that we cannot properly estimate the probability of unobserved answers with the usually-adopted sequence probability calculations. thus, we compute the probability of sequences without sequence length normalization and considering the eos token. formally, we consider the probability of a sequence @xmath4, . . . , t n , eos) as example (revisited). looking back at our previous example, if we consider the eos token in the computation of the probability, we obtain p("vatican"|x) = 0.8 × 0.6 = 0.48, p("vatican city"|x) = 0.8 × 0.4 × 1 = 0.32 and the probability of the unobserved samples is p( ā|x) = 1 -0.48 -0.32 = 0.2. based on this framework, here we present an alternative method for computing the uncertainty of an llm where we directly use the value p( ā|x). we note that this approach, which we call unobserved probability (up), is arguably a very simple way to capture some part of the llm uncertainty, as derived from our analysis. ## experiments and results in this section, we detail our experimental setup to evaluate the relevance of using the probability of unobserved answers for llm uncertainty quantification via our proposed approach up. we compare its performance with three entropy-based methods and also include, for reference, the probability of unobserved answers calculated using the conventional method for sequence probabilities. ## conclusion in this work, we aimed to focus on the probability of unobserved answers, which we note have been overlooked by existing entropy-based llm uq methods. we acknowledge that this probability captures only a portion of the uncertainty. for instance, hesitation between observed answers is not considered since the probability of each separate observed answer is not used. our empirical results are encouraging and in the future we plan to integrate this quantity into existing entropy estimation methods. to achieve this, we believe a theoretical framework that considers both aleatoric and epistemic uncertainty, such as the evidence theory @xcite would be suitable. we also note that current approaches of entropybased uq, present other issues and limitations. although the work of @xcite has made progress in this regard, we think further improvements are necessary, for example, by more directly modeling hypernymy and hyponymy relationships across answers, and/or clusters of answers.