Benchmarking Uncertainty Quantification Methods for Large Language Models with LM -Polygraph (2025)

Abstract: The rapid proliferation of large language models (LLMs) has stimulated researchers to seek effective and efficient approaches to deal with LLM hallucinations and low-quality outputs. Uncertainty quantification (UQ) is a key element of machine learning applications in dealing with such challenges. However, research to date on UQ for LLMs has been fragmented in terms of techniques and evaluation methodologies. In this work, we address this issue by introducing a novel benchmark that implements a collection of state-of-the-art UQ baselines and offers an environment for controllable and consistent evaluation of novel UQ techniques over various text generation tasks. Our benchmark also supports the assessment of confidence normalization methods in terms of their ability to provide interpretable scores. Using our benchmark, we conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.

## introduction uncertainty quantification (uq) is increasingly being recognized as a critical safety component in ai applications. it enables systems to abstain from uncertain model predictions, allowing the associated inputs to be handled through alternative means-for example, by escalating them to a human operator @xcite . this a plethora of uq methods has been developed for classification and regression models @xcite . there has also been a surge of research devoted to uq specifically in the context of encoder-only language models (lms) such as bert @xcite @xcite @xcite @xcite . the rapid proliferation of large language models (llms) has stimulated researchers to seek efficient and effective approaches to uq in text generation tasks, in an attempt to make llms safer to use in downstream applications. as with any machine language (ml) model, llms can make incorrect predictions, ''hallucinate'' by fabricating claims @xcite , or simply generate low-quality outputs. these problems stem from the peculiarities of the llm training objective, the general nature of ml models in being susceptible to errors due to the limited amount of training data, and the inherent ambiguity of tasks. several methods exist for censoring the outputs of llms: output filtering using stop-word lists, post-processing using classifiers @xcite , fact-checking with external tools @xcite , output rewriting @xcite , and model alignment via preference optimization @xcite . however, these techniques alone are insufficient to entirely eliminate incorrect/inappropriate outputs. for instance, fact-checkers target a very narrow sub-problem and usually require external knowledge sources such as knowledge bases, which are generally incomplete. building an efficient external system to verify the llm output for every possible task is infeasible. uq offers a more general solution to the problem by relying on the model's internal capabilities without requiring access to external knowledge, which also enables the potential for greater computational efficiency. several recent studies have focused on developing uq methods for llms in text generation tasks @xcite @xcite @xcite @xcite . however, the current uq research landscape is quite fragmented, with many non-comparable and concurrent studies. researchers have proposed highly divergent methods for benchmarking uq techniques, making it challenging to consolidate research findings and draw general conclusions. in this work, we strive to bridge these disparate research efforts and resolve some issues found in their evaluation protocols by developing a benchmark for uq techniques in text generation tasks. the benchmark is based on the lm-polygraph framework @xcite , which implements state-of-the-art uq baselines in a unified way, enabling a large-scale, consistent comparison of methods developed in recent work. it includes the tasks of selective question-answering (qa), selective generation (machine translation @xcite and text summarization @xcite ), and claim-level fact-checking. for the latter, we developed an automatic fact-checking pipeline for four languages: english, chinese, arabic, and russian. besides common metrics related to uq performance, we also introduce a metric related to the calibration of confidence scores. it enables the evaluation of confidence normalization methods according to their ability to produce interpretable scores. we propose a strong baseline for normalization and investigate its performance in comparison to simpler approaches. using the developed benchmark, we conduct a large-scale empirical investigation of uq and normalization methods across eleven datasets. this work both lowers the barrier to entry into uq research for individual researchers and developers, and enables more robust, reliable, and trustworthy llm deployment for end users. our contributions are as follows: the benchmark can assess the calibration of uncertainty scores and their effectiveness in selective qa/generation and claim-level fact-checking (hallucination detection). foot_0 ## uncertainty normalization methods raw uncertainty scores are good for ranking outputs by their potential quality but can be confusing to the end user. to address this issue, we consider several methods for producing confidence scores bounded within the range [0, 1]. all methods require fitting on a held-out calibration set d @xmath0, y * i } n @xmath1. we assume that for each input x i in this set, output y i is generated by llm, and some uncertainty score u @xmath2, y * i ) are computed. among simple normalization approaches, we consider linear scaling and quantile scaling, as they provide simple rules to normalize uncertainty scores in [0, 1] based solely on uncertainty values u i ; see appendix b for more details. to convey meaningful information about the model's confidence to the end user, the confidence score should not only be bounded within a fixed interval but also directly reflect the expected quality of the model's output. we term this confidence calibration, as opposed to confidence normalization. to achieve this, we introduce two methods referred to as performance-calibrated confidence (pcc). the first approach, binned pcc, splits the calibration set into non-intersecting bins based on the values of uncertainty u i and considers the confidence to be an estimate of the mean quality of the outputs in the bin, as measured by some quality measure of choice. the downside of this approach is that the ordering of the instances based on raw uncertainty and normalized confidence scores can be different, and thus the quality of uq can vary substantially and unpredictably. to address this problem, we propose a second approach: isotonic pcc. it fits centered isotonic regression (cir; oron and flournoy, 2017) on pairs of uncertainty and quality values from the calibration set. cir produces a monotonic piecewise linear function, which allows the use of the relationship between uncertainty and quality while keeping the order of the inputs intact. both approaches in the pcc family produce calibrated confidence scores as a local estimate of some quality measure in the neighborhood of the raw uncertainty estimate. this directly ties the confidence with the estimated quality of the output, thus making it more interpretable than raw uncertainty scores. we provide a more detailed discussion on the specifics of these methods in appendix b. ## approaches to evaluating uncertainty quantification methods in general, a valid uq technique should produce scores that are well correlated with some measure of output quality. thus, the most straightforward way of comparing different uq methods is to measure the rank correlation between some generation quality metric (e.g., rouge-l) and uncertainty scores @xcite . however, this way of evaluating uq is not very informative of the performance gain that a particular uq method achieves. another popular evaluation approach is based on designating outputs as either correct or incorrect based on a threshold over a quality metric, and measuring how well uncertainty scores can predict the output as being one or the other @xcite . this reduces the uncertainty score to being a predictor in a binary classification task, and thus one can use roc-auc or pr-auc as a measure of how well a uq method performs. the problem with this approach, which makes results across different works incomparable, is that it requires selecting the quality threshold, and its choice has been quite arbitrary in the literature. a more comprehensive approach is called rejection verification @xcite . it does not require thresholding the quality metric to formulate a binary classification task. instead, it computes the average quality of the outputs for which the uncertainty is relatively low. by continuously lowering the uncertainty threshold above which data points are discarded, one obtains a set of average quality values of outputs with progressively lower maximum uncertainty. these pairs of uncertainty thresholds and associated average output quality give a prediction-rejection curve. the area under this curve quantifies the overall quality of an uq method @xcite . one problem with the majority of evaluation approaches in previous work on uq for llms is the usage of n-gram output quality metrics such as rouge-l, which often do not reflect the actual quality of the generated output. for example, this discrepancy occurs when the gold standard answer and the llm output differ only by a negation token. in this case, n-gram metrics would rank such a model answer higher than it deserves, failing to capture the substantial semantic difference caused by the negation. in addition to n-gram metrics, we suggest using the recently proposed align-score @xcite , where the gold-standard answer and llm output are compared by another llm. this metric has a higher correlation with human annotators due to its ability to capture deeper semantic similarities and differences between texts. another issue with evaluation protocols in recent works is their tendency to overlook simple yet effective baselines. for instance, msp often proves difficult to surpass in tasks with short outputs. many studies neglect this baseline, favoring comparisons with entropy-based metrics @xcite @xcite , which often perform worse. by not considering this straightforward baseline, the evaluation protocols may give an incomplete picture of uq performance. we also note that research on uq for text fragments, such as sentences and claims, has been limited. furthermore, the evaluation protocols for this setting exhibit significant limitations. for example, @xcite manually annotated texts generated by one llm, then evaluated the uq performance for a proxy model by inferring the probability distributions of the tokens for the fixed output. we argue that such an approach introduces a big discrepancy between the generated text and what a proxy llm actually wants to generate, which results in biased uq performance. in this work, we mitigate this problem by building an automatic evaluation pipeline for uq in claim-level fact-checking for various languages. it allows unrestricted generation from llms and leverages llm-as-a-judge @xcite instead of manual annotation to support experiments with various llms. ## evaluation benchmark our benchmark features three sections: (1) evaluation of uq performance in selective qa/generation; (2) evaluation of uq performance in fact-checking; (3) evaluation of confidence calibration. ## experiments using our benchmark, we evaluated the implemented uq and normalization methods. ## conclusion in this work, we proposed a comprehensive benchmark for evaluating uq techniques in text generation tasks. the empirical investigation conducted on the developed benchmark provides several useful insights. overall, methods based on sample diversity perform well across selective qa/generation tasks. however, for shorter answers, we recommend using information-based methods because they often perform on par with diversity-based techniques, while introducing much less computational overhead. specifically, for multiple-choice qa, information-based methods msp, perplexity, and ccp would be substantially superior. for the tasks that assume longer outputs, methods based on sample diversity such as semantic entropy, degmat, or lexical similarity are preferable. it is worth highlighting that sar consistently stands out as one of the most effective methods for short and long outputs. reflexive methods in general do not demonstrate good performance. only for big llms, such as gpt-4o-mini, it might be reasonable to use verbalized techniques. for the fact-checking task, the best method is ccp: it demonstrates the best results and is computationally efficient. we should also note that msp appears to be a very strong and robust baseline across all tasks and should not be discarded from the evaluation protocols. for generating human-interpretable confidence scores, we suggest normalization based on isotonic pcc as it improves confidence calibration and does not degrade performance in terms of prr. generalizing length-normalized log probability, tokensar @xcite computes the weighted average of the negative log probability of generated tokens based on their relevance for the entire generated text. for a given sentence similarity function g(•, •) and token relevance