Representation-based Broad Hallucination Detectors Fail to Generalize Out of Distribution (2025)

Abstract: We critically assess the efficacy of the current SOTA in hallucination detection and find that its performance on the RAGTruth dataset is largely driven by a spurious correlation with data. Controlling for this effect, state-of-the-art performs no better than supervised linear probes, while requiring extensive hyperparameter tuning across datasets. Out-of-distribution generalization is currently out of reach, with all of the analyzed methods performing close to random. We propose a set of guidelines for hallucination detection and its evaluation.

## introduction while llms @xcite @xcite have made significant progress in various scenarios, they still display undesirable behavior, known as hallucinations, which undermines their reliability in both critical and everyday tasks. hallucinations have been extensively studied, with a wide range of detection methodologies proposed depending on the level of access to model internals. blackbox methods operate on model input-output behavior. these include frameworks verifying generated responses against context @xcite @xcite , evaluating self-consistency across multiple response variants @xcite or multiple verification steps @xcite , querying the model to evaluate the truthfulness of it's own response @xcite or training classifiers to flag inconsistencies @xcite . grey-box methods center on uncertainty quantification through metrics like token-level log probabilities @xcite or entropy of token distribution @xcite . recent white-box methodologies @xcite have focused on detecting hallucinations by leveraging internal representations, moving beyond text-based detection. a line of work learns to predict the hallucination by training linear probes on the hidden states of llms directly @xcite or to approximate derived metrics like semantic entropy @xcite or semantic consistency @xcite . among white-box methods, many recent approaches rely on analyzing the attention mechanisms of transformer models, which has garnered significant research interest. working under the assumption that retrieval heads @xcite are an important mechanism for information propagation within @xcite show that masking the retrieval heads leads to increased hallucination incidence and that contrasting the next-token predictions of the base model with those of the masked one can act as a mitigation mechanism. @xcite further explores this attention-based methodology, which we analyze in depth in section 2. treating all types of hallucinations as a single category is challenging, resulting in methods that perform well in some scenarios (e.g., hallucination detection in summarization tasks) but poorly in others. retrieval-augmented generation (rag) has gained traction as a method to potentially decrease hallucinations. nevertheless, even with rag, models can still fail to correctly attend to the context or overwrite it with parametric knowledge. due to the limited availability of human-labeled rag hallucination datasets, studying hallucinations related to contextual errors is difficult, with only a few attempts so far @xcite @xcite . we focus on the current sota approaches for detecting hallucinations using internal representations, with the goal of defining a clearer future trajectory for hallucination detection. our contributions are as follows. (a) we verify hallucination detectors based on model internals via linear probes, a random forest classifier and sparseautoencoder (sae) probes. (b) assess sota and find that most of its performance is due to spurious correlation rather than genuine hallucinatory signal. (c) analyze detection out-of-distribution and find that generalization remains a challenge. (d) find that sae features do not provide consistent benefits for hallucination detection. ## state of the art redeep @xcite employs internal model mechanisms to identify hallucinations, specifically those cases where external context serves as the ground truth but may be compromised by the model's parametric knowledge. it relies on copying heads @xcite and assumes that attention scores are an appropriate proxy for what information the model integrates from the context. moreover, the parametric interventions of the model, measured by the jensen-shannon divergence between pre-and post-mlp representations, serve as an important hallucination indicator. they hypothesize that increased attention scores correlate with a decrease in the hallucination incidence, while elevated parametric interventions correspond to increased occurrence of hallucinations. redeep comes in two flavors: token-and chunk-based, where the attention and parametric scores are measured on individual tokens or token chunks, respectively. redeep achieves sota hallucination detection results on two publicly available rag hallucinations datasets: ragtruth and dolly (ac). azaria and mitchell (2023) introduce saplma, another representation-based hallucination detection method. they investigate whether the llm possesses some notion of truthfulness of generated statements in its hidden state. the authors postulate that if it does, this information can be used to detect and mitigate hallucinations. based on experimental results, they hypothesize that the hidden states indeed encode signals of truth or falsehood, leading to the hypothesis that the model may "know" when it hallucinates. saplma performs on-par with redeep in hallucination detection on ragtruth. ## spurious correlation explains sota performance redeep's evaluation focuses exclusively on the performance across all three subtasks in the ragtruth dataset. this prompts us to investigate its performance on individual tasks. the fraction of model responses which human annotators labeled as hallucinations (hallucination rate) is notably higher for the d2t task ( we construct a naïve classifier which follows a heuristic: it predicts 1 (i.e., hallucinated sample) if the input's task type is d2t, and 0 for all other tasks. in the d2t task is highly specific as it relies on the eval dataset column specifies the dataset on which the method is evaluated, whereas the hyper-params column indicates the dataset used to fine-tune the hyperparameters of the method. therefore, detectors trained and tested on the entire ragtruth dataset may in reality respond to the presence of json in the prompt. to verify that detecting the type of the task from model activations is possible, we collect model activations from the last token and the last layer of the llama2-7b-chat model and we use them to train a logistic regression to predict the json task. the trained linear probe achieves perfect prediction on the test set, with an auc of 1.0. the evaluation problems are not an issue with the ragtruth dataset itself. in fact, the results in the dataset paper @xcite are reported per task. this distinction also exists in methods proposed in @xcite @xcite . however, aggregated metrics have also been reported in @xcite . ## hallucination detection with model internals the fundamental question we should address first is this: is it possible to classify a response from an llm as hallucinatory based on its internal states? if this is possible, then it would be a natural benchmark against other methods based on model internals. to provide an answer, we extract the activations of the llm in the prompt processing and answer generation phases and then use those activations as input to a classifier. extraction is performed from the residual stream: pre-attention (resid_pre) and pre-mlp (resid_mid). we consider linear and non-linear probes (raw activations, details in appendix d), and sae probes (sae features, details in appendix d.1). ## conclusion we have assessed general hallucination detectors and found that sota performance on the ragtruth dataset may be overstated due to evaluation problems. on top of that, even in more rigorous settings, sota is often outperformed indistribution by linear probes. furthermore, each of the considered methods performs no better than random out-of-distribution. this underscores the challenge in general hallucination detection, where current methods based on model internals are unlikely to generalize to unseen data. in light of this challenge, we propose a set of guidelines for future hallucination detection methods, in particular: restricting the scope of the hallucination and focusing on specific hallucination spans rather than general labels -potentially promising angles of attack.