FACTCHECKMATE : Preemptively Detecting and Mitigating Hallucinations in LM s (2025)

Abstract: Language models (LMs) hallucinate. We inquire: Can we detect and mitigate hallucinations before they happen? This work answers this research question in the positive, by showing that the internal representations of LMs provide rich signals that can be used for this purpose. We introduce FactCheckmate, which preemptively detects hallucinations by learning a classifier that predicts whether the LM will hallucinate, based on the model’s hidden states produced over the inputs, before decoding begins. If a hallucination is detected, FactCheckmate then intervenes by adjusting the LM’s hidden states such that the model will produce more factual outputs. FactCheckmate provides fresh insights that the inner workings of LMs can be revealed by their hidden states. Practically, both its detection and mitigation models are lightweight, adding little inference overhead; FactCheckmate proves a more efficient approach for mitigating hallucinations compared to many post-hoc alternatives. We evaluate FactCheckmate over LMs of different scales and model families (including Llama, Mistral, Qwen and Gemma), across a variety of QA datasets from different domains. Our results demonstrate the effectiveness of FactCheckmate, achieving over 70% preemptive detection accuracy. On average, outputs generated by LMs with intervention are 34.4% more factual compared to those without.

## introduction language models (lms) hallucinate, a phenomenon where they produce nonfactual or even misleading outputs that often appear plausible @xcite @xcite @xcite @xcite . extensive efforts have been devoted to mitigating their hallucination issues * equal contribution. @xcite @xcite . these approaches are mostly reactive, addressing hallucinations after they occur, and often require resampling new outputs @xcite , substantially increasing the inference overhead. in addition, they often treat the lm as a black box, while relying on external lms for detecting hallucinations, missing the opportunity to gain deeper insights into the internal workings of these models. recent findings by azaria and mitchell (2023) and @xcite show that the lms' representations can provide useful information about the factuality of their outputs. @xcite observe that lms' hidden states generated over factual and non-factual statements are linearly separable. however, these studies have a relatively narrow focus, primarily addressing hallucination detection in a reactive manner, and a more thorough investigation is needed. the key hypothesis of this paper is that, the lms' hidden states reveals valuable information about their internal working mechanisms, and provide signals that can be used to predict whether it will hallucinate before decoding. we propose factcheck-mate to answer the following research question (rq): can we preemptively predict and mitigate hallucinations with lms' internal representations? factcheckmate learns a classifier that, taking the models' hidden states over the inputs, predicts whether the model is about to hallucinate. if a hallucination is detected, factcheckmate intervenes, by adjusting the lm's hidden states with a learned intervention model, and steering them towards producing more factual outputs (figure 1 ). our controlled experiments answer the rq in the positive. we evaluate factcheckmate across four qa datasets from different domains: nq-open (wikipedia; @xcite , mmlu (stem exams; @xcite , medmcqa (medical; @xcite , and gsm8k (math; cobbe et al., 2021). using the lms' hidden states over the input questions, factcheckmate can successfully predict whether or not the lms will hallucinate over 70% of the time, significantly outperforming a 50% random baseline ( §3). we observe consistent trends across base and fine-tuned lms of different scales and families, including llama2 (7b and 13b; touvron et al., 2023a), llama3 (8b and 70b) and llama3.1 (8b and 70b) (dubey et al., 2024) mistral-7b (jiang et al., 2023), gemma-7b (team et al., 2024), and qwen2.5 (7b and 32b) qwen et al., 2025. we further conduct cross-model and crossdataset evaluations to assess the generalizability of factcheckmate. in the cross-model training, we train on the hidden states of multiple models. whereas, in the cross-dataset, we train the classifier on multiple domains. the cross-model classifier achieves on average 71% accuracy in preemptive hallucination detection, demonstrating its robustness across different models. similarly, the cross-dataset classifier attains an average of 65% accuracy. furthermore, factcheckmate's intervention model can effectively improve the lms' factuality. using gpt-4o as a judge, which shows high agreement with human evaluations in our experiments, we find that on average, outputs generated by lms with intervention are 34.4% more factual than those produced without intervention ( §4). we also calculate the inference time overhead introduced by factcheckmate, incurring minimal average overhead of a 1.2% increase in decoding time, showing minimal impact on inference efficiency ( §5.1). factcheckmate reveals surprising insights into existing lms, and can potentially lead to more profound understanding of their internal working. all code, data, and checkpoints for reproducing our findings will be released. existing research primarily focuses on postprocessing methods applied after the inference process is completed and often utilizing external knowledge for verification @xcite @xcite . for instance, critic @xcite validates model outputs through tool interactions, and factscore @xcite breaks down generated content into atomic facts, assessing their accuracy by comparing them against reliable sources. a recent line of research leverages the internal mechanics of lms to detect hallucinations @xcite @xcite . @xcite locates where factual associations are stored in gpt models. these studies have spurred further research into using lms' internal representations in hallucination detection @xcite . for instance, mind @xcite generates training data in unsupervised approach for training hidden states based hallucination detectors. duan et al. (2024) conducts an experimental examination of the hidden states of llms when processing factual versus nonfactual responses. factcheckmate demonstrates the effectiveness of preemptive hallucination detection, identifying warning signals several tokens before the hallucinations actually occur, via the language model's hidden states. hallucination mitigation. for hallucination mitigation at inference time, existing works have explored self-correction and automated feedback approaches, where the language model is prompted to fix its generation flaws, with or without leveraging feedback from the model itself or some external knowledge source @xcite @xcite . a recent approach involves utilizing activation engineering @xcite @xcite . factcheckmate builds on these findings and explores activation engineering techniques to preemptively intervene and mitigate hallucinations during inference time. it is also related to inferencetime approaches that utilize a scoring function to steer the lm toward desired behavior @xcite . hidden states as predictive signals of hallucination. previous works primarily use hidden states as indicators of factuality after generation @xcite @xcite . factcheckmate instead asks: can hidden states reveal early signals of hallucination before tokens are generated? if so, this would suggest that factuality cues are embedded in the model's internal mechanisms earlier than previously assumed. factcheckmate demonstrates, for the first time, that hidden states offer early signals correlated with hallucination. its finding reveals that factuality cues are embedded within the model's internal mechanisms well before the output is generated. this fresh insights allows factcheckmate to use the model's hidden states to anticipate when it is likely to hallucinate, rather than waiting for errors to surface in generated tokens. this reduces the need for expensive post-hoc corrections and provides insight into how factual knowledge is internally encoded and accessed. our results ( §3) suggest that hallucinations are not merely failures of token-level prediction but often emerge from systematic patterns in how models encode factual information during inference @xcite . ## preemptive hallucination detection this section focuses on factcheckmate's preemptive hallucination classifier ( §3.1) and experimental results ( §3.2). ## factcheckmate preemptive hallucination mitigation this section focuses on using factcheckmate to preemptively mitigate hallucinations, including its intervention model ( §4.1) and experiments ( §4.2). ## additional experiments in the following section, we first evaluate the inference time overhead ( §5.1). next, we investigate the role of word embedding layers ( §5.2). other experiments are detailed in ( §h) ## conclusion in conclusion, factcheckmate demonstrates that the lms' hidden states encode rich information that can be used to predict hallucination preemptively, even before they appear in the generated output. leveraging this insight, we develop a preemptive detection and intervention mechanism that steers the lm's generation towards more factual outputs, once the hallucination is likely to occur. we achieve a preemptive hallucination detection accuracy of more than 70%, and an average of 34.4% more factual output by lms supported by factcheckmate, compared to the base lms. we empirically prove the significant potential of utilizing the internal working of lms, through learning lightweight models for hallucination detection and mitigation, introducing a negligible average ties base wins while our current experiments focus on qa tasks due to their structured evaluation, the core idea behind factcheckmate leveraging hidden states for preemptive hallucination detection and mitigation could extend to other generation tasks. for tasks like summarization or dialogue systems, we anticipate similar patterns in hidden states when the model is about to generate hallucinated content. however, pinpointing where the hallucination starts to analyze preceding hidden states is challenging and presents a more nuanced problem. therefore, alternative assessment metrics, such as faithfulness scoring or costly human annotation, would be needed instead of exact match evaluation. additionally, we have only looked at the hidden states as an internal component for classification to predict the factuality of a sentence. exploring other lm's internal components presents a potential direction for future work. this pipeline also would not work for black-box lms.