Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification (2024)

Abstract: Large language models (LLMs) are notorious for hallucinating, i.e., producing erroneous claims in their output. Such hallucinations can be dangerous, as occasional factual inaccuracies in the generated text might be obscured by the rest of the output being generally factually correct, making it extremely hard for the users to spot them. Current services that leverage LLMs usually do not provide any means for detecting unreliable generations. Here, we aim to bridge this gap. In particular, we propose a novel fact-checking and hallucination detection pipeline based on token-level uncertainty quantification. Uncertainty scores leverage information encapsulated in the output of a neural network or its layers to detect unreliable predictions, and we show that they can be used to fact-check the atomic claims in the LLM output. Moreover, we present a novel token-level uncertainty quantification method that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use. Our method Claim Conditioned Probability (CCP) measures only the uncertainty of a particular claim value expressed by the model. Experiments on the task of biography generation demonstrate strong improvements for CCP compared to the baselines for seven different LLMs and four languages. Human evaluation reveals that the fact-checking pipeline based on uncertainty quantification is competitive with a fact-checking tool that leverages external knowledge.

## introduction large language models (llms) have become a ubiquitous and versatile tool for addressing a variety of natural language processing (nlp) tasks. people use these models for tasks including information search @xcite , to ask medical ♢ equal contribution questions @xcite , or to generate new content @xcite . recently, there has been a notable shift in user behavior, indicating an increasing reliance on and trust in llms as primary information sources, often surpassing traditional channels. however, a significant challenge with the spread of these models is their tendency to produce "hallucinations", i.e., factually incorrect generations that contain misleading information @xcite . this is a side-effect of the way modern llms are designed and trained @xcite . llm hallucinations are a major concern because the deceptive content at the surface level can be highly coherent and persuasive. common examples include the creation of fictitious biographies or the assertion of unfounded claims. the danger is that a few occasional false claims might be easily obscured by a large number of factual statements, making it extremely hard for people to spot them. as hallucinations in llm outputs are hard to eliminate completely, users of such systems could be informed via highlighting some potential caveats in the text, and this is where our approach can help. fact-checking is a research direction that addresses this problem. it is usually approached using complex systems that leverage external knowledge sources @xcite @xcite . this introduces problems related to the incomplete nature of such sources and notable overhead in terms of storing the knowledge. we argue that information about whether a generation is a hallucination is encapsulated in the model output itself, and can be extracted using uncertainty quantification (uq) @xcite @xcite . this avoids implementing complex and expensive factchecking systems that require additional computational overhead and rely on external resources. prior work has mainly focused on quantifica-tion of uncertainty for the whole generated text and been mostly limited to tasks such as machine translation @xcite , question answering @xcite , and text summarization @xcite . however, the need for an uncertainty score for only a part of the generation substantially complicates the problem. we approach it by leveraging token-level uncertainty scores and aggregating them into claim-level scores. moreover, we introduce a new token-level uncertainty score, namely claim-conditioned probability (ccp), which demonstrates confident improvements over several baselines for seven llms and four languages. to the best of our knowledge, there is no previous work that has investigated the quality of claim-level uq techniques for llm generation. therefore, for this purpose, we construct a novel benchmark based on fact-checking of biographies of individuals generated using a range of llms. note that different llms produce different outputs, which generally have higher variability than, e.g., outputs in such tasks as machine translation or question answering. therefore, we compare the predictions and uncertainty scores to the results of an automatic external fact-checking system factscore @xcite . human evaluation verifies that our constructed benchmark based on factscore can adequately evaluate the performance of the uncertainty scores. our contributions are as follows: all the code and data for experiments is publicly available foot_0 . ## fact-checking pipeline the fact-checking pipeline (see figure 4 ) starts with splitting a generated text into atomic claims, e.g. using a much smaller model fine-tuned for this particular task. for experimental evaluations in this work, we follow the factscore approach @xcite , where splitting is implemented via the openai chat api. each atomic claim is matched against the sequence of tokens in the original text with the corresponding probability distributions. then we calculate token-level uncertainty scores and aggregate them into the claim-level uncertainty. finally, the claim-level uncertainty scores are compared against a threshold obtained on a validation set to determine whether the claim should be highlighted for the end-user as unreliable. individual tokens can be a part of multiple atomic claims. if the token belongs to a reliable and unreliable claim at the same time, it is not highlighted. an example visualization is presented in figure 1 . ## uncertainty quantification in this section, we first provide background on common uq methods that can be used at the token level, sentence: she attended the school of design, where she earned a bachelor of fine arts degree in painting. ^^^^^^^^ ^^ ^^^^ ^^^^ ^^^^^^ ^^ ^^^^^^^f act: she earned a bachelor of fine arts degree in painting. she attended the school of design, where she earned a bachelor of fine arts degree in ... then delve into our claim-conditioned probability (ccp) token-level method, and finally describe how token-level uncertainties are aggregated into a claim-level score. autoregressive language models generate text token by token. in this work, we will operate on the level of words and without loss of generality suppose that the autoregressive distribution at each step generates the random word x j ∼ p (• | @xmath0, where @xmath1. we also denote by x j the generated word at position j and by x 1:@xmath2. for example, in the case of greedy generation, x @xmath3, where x j is the most probable realization of x j . let us also denote by c a set of indices of words corresponding to a particular atomic claim. ## benchmark for evaluation of claim-level uq methods we evaluate claim-level uq techniques and their ability to spot hallucinations on the task of generating biographies. in relevant previous work @xcite , the authors generate biographies with one llm (gpt-3), manually annotate sentences for factuality, and quantify uncertainty of a different "proxy" model. factuality labels are then used to evaluate the quality of uncertainty scores. we argue that such an approach based on a proxy model introduces a big discrepancy between the generated text and what a proxy llm actually wants to generate, which results in biased uq evaluation results. to make the evaluation as close as possible to the real-world scenario, we allow unrestricted generation of biographies from llms. unrestricted generation complicates automatic evaluation of the fact-checking pipeline, because obtaining gold standard annotation requires manual annotation of all outputs from each model. therefore, in addition to manual annotation, we annotate claims in generated texts automatically using factscore -a fact-checking tool @xcite , which has access to an external knowledge source. using factscore, enables completely automatic evaluation and allows us to scale up experiments. we generate llm responses in english, chinese, arabic, and russian to 100 biography prompts. the typical biography prompt is give me a biography @xmath4. the set of people was generated by asking gpt-4 to list the most famous people since 1900. the maximum generation length is set to 256 tokens. if the last sentence of the generation is unfinished (i.e. does not end with any punctuation), it is discarded. we generate responses for the following llms: (for english) vicuna 13b @xcite , mistral 7b @xcite , jais 13b @xcite , and gpt-3.5-turbo @xcite ; (for chinese) yi 6b @xcite ; (for arabic) jais 13b and gpt-4; (for russian) vikhr-instruct-0.2 7b @xcite . we decompose the generated text into atomic claims using gpt-4. for each claim, we map all its words back to generated text to access the corresponding token logits. not all claims perfectly match to the original response. for example, for vicuna 13b around 5% of all claims do not successfully match because chatgpt abstained to respond or outputted words not present in the original. we consider only successfully matched claims. for english, we are able to perform annotation completely automatically: each atomic claim is classified by factscore as supported or not supported. the underlying factscore model is "retrieval+chatgpt" with a dump of wikipedia articles as an external knowledge source. we also manually annotated 100 claims from english biographies produced by the vicuna 13b model, 183 claims from arabic biographies, 146 claims from russian biographies, and 1603 claims in chinese. each of the statements was checked by two annotators with access to the corresponding wikipedia article. the final label is set to "supported" only if both annotators label it as so. the statistics of the resulting datasets with ## conclusion we presented a novel approach to fact-checking and hallucination detection based on token-level uncertainty quantification. according to human evaluation, our approach is competitive with factscore, a fact-checking tool that leverages an external knowledge source: we achieve similar or better results with access to only llm outputs. we proposed a computationally efficient tokenlevel and claim-level uq method, claim conditioned probability that outperforms a number of baselines in fact-checking. in this method, we postprocess the word distribution to mitigate the impact of uncertainty related to the variability of surface forms and uncertainty about what claim type to generate on the current step. in the constructed benchmark, where we detect hallucinations in biographies, ccp outperforms other methods for seven llms, including gpt-3.5-turbo and gpt-4, and four languages. we also demonstrate that computational overhead of ccp might be as low as 3% of the llm inference runtime.