Embedding and Gradient Say Wrong: A White-Box Method for Hallucination Detection (2024)

Abstract: In recent years, large language models (LLMs) have achieved remarkable success in the field of natural language generation. Compared to previous small-scale models, they are capable of generating fluent output based on the provided prefix or prompt. However, one critical challenge — the *hallucination* problem — remains to be resolved. Generally, the community refers to the undetected hallucination scenario where the LLMs generate text unrelated to the input text or facts. In this study, we intend to model the distributional distance between the regular conditional output and the unconditional output, which is generated without a given input text. Based upon Taylor Expansion for this distance at the output probability space, our approach manages to leverage the embedding and first-order gradient information. The resulting approach is plug-and-play that can be easily adapted to any autoregressive LLM. On the hallucination benchmarks HADES and other datasets, our approach achieves state-of-the-art performance.

## introduction in recent years, large language models, such as @xcite , llama @xcite and palm @xcite , have achieved tremendous successes. llms generate fluent output based on given texts and prompts. they have been applied in various real-world scenarios, such as dialogue systems and information retrieval systems @xcite . however, despite their powerful generative capabilities, llms still suffer from the problem of hallucinations: given an input, the model generates text unrelated to the source or even contradictory @xcite . even the most advanced language model currently available, gpt-4, has been shown to experience hallucinations @xcite . these hallucination issues severely compromise the fairness and safety of llms, hindering their large-scale deployment. indeed, to solve this issue, the task of hallucination detection, aiming to determine whether hallucination occurs, has attracted more and more attention from the community. the existing methods for hallucination detection primarily focus on detecting at the token level in the output @xcite . some methods utilize uncertainty metrics at the token level to determine hallucination @xcite , while some other methods design prompts to query the model multiple times to assess the hallucination @xcite . despite the prior efforts, we believe that white-box hallucination detection is not much studied. the white-box detection setting can both benefit the open-sourced llm community -such as llama or mistral @xcite -and prompt the llm owners to provide extra services upon the current forms of api. contrary to the black-box setting, a white-box setup may utilize the full token probability matrices and the embedding layers in the transformer. in this work, we dive into this white-box setup and prove that this internal information may significantly benefit detecting hallucinations. however, given the current parameter scale of the llm, devising a white-box detector is nontrivial. this means we need to systematically determine what portion of the information in the llm is discriminative to raise the hallucination case. the motivation for this study stems from a rather simple and straightforward idea, as follows. we translate the verbal definition of hallucination to be more formal. take the qa task as an example, where q and a represent the question and the model's answer, respectively. p (a|q) and p (a) represent the probability distributions of the model outputting a under conditional q and unconditional scenarios. the hallucination detection task can be considered as a task to determine how much the model answer a depends on the question q, that is, modelling the difference between p (a) and p (a|q) as denoted as d(p (a), p (a|q)). indeed, one can directly use metrics like kl divergence, cross-entropy or others to calculate the difference between p (a) and p (a|q). however, these measurements would only offer a singledimensional output as the indicator, which we found quite imprecise towards the detection of hallucination. we argue that simply using the probability space for this problem overlooks the complexity of the hallucination issue due to the massive amount of information wasted and dropped. in our approach, by contrast, we first conduct taylor series of d(p (a), p (a|q)). we further show that the subitems from the resulting series can be modelled by the embeddings in the transformer combined sophistically with the first-order gradient information. this result paves the way to effectively detect hallucinations. figure 1 provides a visual example of our approach, named egh (embedding and gradientbased hallucination detection method). more specifically, inspired by filippova (2020), we construct a dual form, (i)-the conditional tuple jointing the source-output text versus (ii)-its unconditional counterpart with only the output text. after feeding both forms through the llm, we yield the d(p (a), p (a|q)) mentioned above. we cache the corresponding necessary information, including embedding and gradient, alongside the two feedforward passes. on top of them, we train a simple classifier to discriminate if the hallucination occurs. our method is a model-agnostic solution for white-box hallucination detection, which relies on model internal signals (embedding and gradient information) to perform. in order to make our method solid, we managed to experiment with multiple llms as base models. on several common hallu-cination detection datasets, including halueval @xcite , selfcheckgpt @xcite , and hades @xcite , egh establishes a set of state-of-the-art performances. ## conclusion this paper proposes a white-box hallucination detection method named egh that utilizes the internal embedding and gradient of the model to determine hallucination. egh is based on the following assumptions: during hallucination generation, the model tends to generate responses without considering the input question directly. the model's understanding of the input question represents the degree of hallucination. we designed both conditioned and unconditioned inputs and utilized the taylor expansion method to demonstrate that embeddings and gradient features can represent this degree. egh achieves state-of-the-art results on hallucination detection datasets such as halueval, selfcheckgpt, and hades, validating the effectiveness of our method. our work provides valuable insights into detecting hallucinations in llm.