Decoding Uncertainty: The Impact of Decoding Strategies for Uncertainty Estimation in Large Language Models (2025)

Abstract: Decoding strategies manipulate the probability distribution underlying the output of a language model and can therefore affect both generation quality and its uncertainty. In this study, we investigate the impact of decoding strategies on uncertainty estimation in Large Language Models (LLMs). Our experiments show that Contrastive Search, which mitigates repetition, yields better uncertainty estimates on average across a range of preference-aligned LLMs. In contrast, the benefits of these strategies sometimes diverge when the model is only post-trained with supervised fine-tuning, i.e. without explicit alignment.

## introduction recent advances in natural language processing (nlp) have been driven almost entirely by the rapid progress of large language models (llms). state-of-the-art models such as @xcite , llama @xcite already match or surpass human performance on a diverse suite of downstream nlp tasks. despite these successes, llms sometimes output fabricated or misleading text (hallucinations), which hinders the deployment of llms in safetycritical domains such as medicine, finance, and law. uncertainty estimation (ue) is a key technique for mitigating the problem @xcite @xcite . by quantifying predictive uncertainty, a system can reject dubious outputs and route them to either human experts or stronger models. in addition, decoding strategies also constitute a promising approach to address the problem. decoding strategies manipulate next-token distributions of language models, thereby able to elicit higher-quality outputs from the language model. recent work demonstrates that the choice of decoding strategy can markedly impact the quality of llm outputs, underscoring its pivotal role in unlocking the full potential of these models @xcite . however, the comprehensive investigation into how decoding strategies affect ue performance in llms remains limited. although recent studies improve uncertainty by devising sampling strategies @xcite , the ue performance combined with extensive decoding strategies has not been systematically evaluated across various tasks. since decoding algorithms influence both the probability distribution over candidate tokens and the final token selection, they can have a significant impact on ue performance. furthermore, mainstream llms usually apply preference-alignment techniques including reinforcement learning from human feedback (rlhf) @xcite or direct preference optimization (dpo) @xcite after supervised fine-tuning (sft). although such techniques improve the alignment of outputs with human preferences, recent work suggests they can degrade reliability @xcite @xcite , potentially interacting with the choice of decoding strategy. the investigation of these interactions is therefore essential for producing more trustworthy llm outputs. we address this gap through two research questions: our experiments reveal the following findings: first, contrastive search, which explicitly mitigates repetition, achieves better ue performance as a whole. second, the optimal decoding strategy for ue can change as a model pro-gresses from sft to preference alignment during its post-training phase. in addition, our results show that the changes in ue performance depend on the interaction with the decoding strategy and preference-alignment techniques. all code is available at https://github.com/ wataruhashimoto52/decoding_uncertainty . ## decoding strategies we focus exclusively on deterministic decoding strategies because deterministic outputs are important in safety-critical domains such as finance @xcite . the strategies examined in this study are: greedy search (greedy), beam search (bs) @xcite , diverse beam search (dbs) @xcite , contrastive search (cs) @xcite , contrastive decoding (cd) @xcite , frustratingly simple decoding (fsd; based on an n-gram model) @xcite , fsd-vec (based on a vectorized n-gram model), decoding by contrastive layers (dola) @xcite , and self-logits evolution decoding (sled) @xcite . technical details and the hyper-parameter search space are provided in appendix a. ## conclusion in this study, we examined how decoding strategies affect predictive uncertainty in llms. our experiments show that contrastive search strategy tends to provide better uncertainty estimates across various tasks and models on average by mitigating output repetition, a key source of model overconfidence. on the other hand, we found that the conclusions may change depending on the stage in the post-training phase, such as sft and the preference-alignment. we hope that this study will help practitioners improve the reliability of llms.