Certain but not Probable? Differentiating Certainty from Probability in LLM Token Outputs for Probabilistic Scenarios (2025)

Abstract: Reliable uncertainty quantification (UQ) is essential for ensuring trustworthy downstream use of large language models, especially when they are deployed in decision-support and other knowledge-intensive applications. Model certainty can be estimated from token logits, with derived probability and entropy values offering insight into performance on the prompt task. However, this approach may be inadequate for probabilistic scenarios, where the probabilities of token outputs are expected to align with the theoretical probabilities of the possible outcomes. We investigate the relationship between token certainty and alignment with theoretical probability distributions in well-defined probabilistic scenarios. Using GPT-4.1 and DeepSeek-Chat, we evaluate model responses to ten prompts involving probability (e.g., roll a six-sided die), both with and without explicit probability cues in the prompt (e.g., roll a fair six-sided die). We measure two dimensions: (1) response validity with respect to scenario constraints, and (2) alignment between token-level output probabilities and theoretical probabilities. Our results indicate that, while both models achieve perfect in-domain response accuracy across all prompt scenarios, their token-level probability and entropy values consistently diverge from the corresponding theoretical distributions.

## introduction as large language models (llms) are increasingly integrated into decision-support and knowledgeintensive applications, uncertainty quantification (uq) is essential to ensure reliable downstream use @xcite . prior work has focused on leveraging the token logitsnumerical representations encoding the model's output probabilities-for uq methods applied to natural language generation tasks @xcite @xcite @xcite @xcite . token logits can be transformed into probabilities using activation functions (e.g., softmax or sigmoid), enabling entropy computation over the token distribution. these probability and entropy values are often used to quantify model certainty, providing token-level or response-level confidence scores to users. for prompts involving randomness, risk, or chance, traditional uq alone may be insufficient to capture the true confidence a user should have in implementing a model. key insight: specifically, for prompts involving probabilistic scenarios with inherent aleatoric uncertainty (e.g., "flip a fair coin"), a model's behavior is trustworthy only when its distribution over possible outputs matches the intended theoretical distribution (which may be only implicitly specified). hence, in these probabilistic scenarios, we argue there are two considerations for uncertainty quantification that are particularly important: these considerations are somewhat in contrast to traditional uq settings, where model accuracy typically corresponds to low uncertainty when a model is well-calibrated. in this work, we explore the relationship between token certainty and theoretical probability in welldefined probabilistic scenarios. we prompt gpt-4.1 and deepseek with 10 well-known scenarios (e.g., roll a six-sided die or pick a card from a deck ## experimental results for both the unspecified and specified prompt sets, we generate five independent samples from each model. we compute the mean token-level probability and entropy values across these samples and compare the aggregated statistics to the theoretical values defined by the corresponding probabilistic scenario (listed in we find that the majority of samples have more than one response in common except for gpt-4.1's bingo (unspecified and specified) and month and day (specified) samples. for the coin-flip and dice-roll scenarios, both deepseek-chat and gpt-4.1 produce the same outcome across all samples ("heads" and "4", respectively), and in the rock-paper-scissors scenario, both models predominantly select "scissors." the most frequent responses differ depending on the model and prompt for all other scenarios. all responses from both models are within the probabilistic scenario constraints, with the exception of deepseek-chat providing partial responses for bingo and roulette. for example, gpt-4.1 specifies the roulette pocket with both color and number (e.g., 27 black), whereas deepseek-chat provides only the number (e.g., 14). similarly, in the bingo scenario, gpt-4.1 provides the letter and number (e.g., g-52) but deepseek only provides a number (e.g., 17). we consider these partial responses to be accurate within the scenario as the numeric values are valid options. ## discussion because llms are considered effective tools for individual tasks or as components within larger processing pipelines, uq is particularly important, not only to obtain confidence estimates, but also to understand precisely what those estimates represent. in some use cases, it may be necessary to expand uq methods to capture task-specific requirements more comprehensively. in this work, we examine uq in the context of prompts involving probabilistic scenarios, where an optimal model output would align with the corresponding theoretical probability distributions. our experiments show that both deepseek-chat and gpt-4.1 fail to achieve this alignment, specifically in straightforward cases such as a coin toss or die roll. despite their failure to achieve probabilistic calibration, both deepseek-chat and gpt-4.1 attained 100% accuracy in response validity across all samples, demonstrating consistent in-domain knowledge of the probability-oriented tasks. however, their response certainty levels varied by scenario. for example, in the unspecified prompts, both deepseek-chat and gpt-4.1 exhibited the highest certainty on the coin flip task (pr(t) = 1.0 and h(t ) ≤ 0.0002 for the token "heads"). deepseek-chat additionally produced a highly confident response in the shakespeare scenario (pr(t) = 0.979 and h(t ) = 0.144 for the token "hamlet"). in contrast, the greatest response uncertainty was observed in the bingo (pr(t) = 0.09 and h(t ) = 2.9), dart on map (pr(t) = 0.227 and h(t ) = 3.47), and roulette (pr(t) = 0.13 and h(t ) = 3.39) scenarios for gpt-4.1, and in the die roll (pr(t) = 0.3; h(t ) = 1.24) and roulette (pr(t) = 0.08 and h(t ) = 2.04) scenarios for deepseek-chat. our findings are consistent with @xcite , even with our difference in prompt style; we did not explicitly specify the constraints of the probabilistic scenario, but rather experimented with well-known scenarios. additionally, our prompts elicited responses consisting solely of the final answer, ensuring that the observed biases are attributable exclusively to token-level probabilities. in typical uq settings, certainty measurements must account for the auto-regressive generation process of llms; however, by removing this factor in our experiments, we demonstrate that systematic biases persist even in the absence of sequential generation effects. motivated by our findings, we explore three additional questions for deeper examination and discussion: sure! let's flip a virtual coin: result: heads (it's a 50/50 chance-refresh if you'd like to flip again!) scenario: coin flip a coin. scenario: roulette certainly! an american roulette wheel has 38 pockets: numbers 1 through 36, plus a single zero (0) and a double zero (00). here's your spin result: the ball lands in pocket: 17 (red) (note: this is a randomly generated result for simulation purposes. in a real game, the outcome is entirely random.) would you like to spin again? 🎲 spin an american roulette wheel. tell me which pocket it lands in. ulate the correct probability distributions for the prompted scenarios, reflecting accurate reasoning and in-domain knowledge. despite their dialogue responses, deepseek-chat and gpt-4.1 generated response tokens with probability and entropy values that did not align with the corresponding theoretical distributions. this discrepancy between verbalized reasoning and token-level sampling surfaces a critical gap between model reasoning explanations and probabilistic calibration. ## conclusion the divergence between certainty and probability has important implications for deploying llms in high-stakes decision-support contexts, where probabilistic calibration is critical for trustworthy system behavior. in this study, we examined the relationship between token-level certainty and theoretical probability alignment in llms, focusing on probabilistic scenarios with well-defined distributions. across ten probability-oriented prompts, gpt-4.1 and deepseek-chat consistently generated valid responses within scenario constraints; however, their token-level probability and entropy values differed from the corresponding theoretical distributions. these results highlight an important distinction in uncertainty quantification between response certainty and probabilistic calibration. additional evaluation is required when alignment with a theoretical probability distribution is a critical aspect of the task. without such assessment, a model's apparent accuracy may mask deficiencies in its probabilistic calibration.