SAFE : A Sparse Autoencoder-Based Framework for Robust Query Enrichment and Hallucination Mitigation in LLM s (2025)

Abstract: Despite the state-of-the-art performance of Large Language Models (LLMs), these models often suffer from hallucinations, which can undermine their performance in critical applications. In this work, we propose SAFE, a novel framework for detecting and mitigating hallucinations by leveraging Sparse Autoencoders (SAEs). While hallucination detection techniques and SAEs have been explored independently, their synergistic application in a comprehensive system, particularly for hallucination-aware query enrichment, has not been fully investigated. To validate the effectiveness of SAFE, we evaluate it on two models with available SAEs across four diverse cross-domain datasets designed to assess hallucination problems. Empirical results demonstrate that SAFE consistently improves query generation accuracy and mitigates hallucinations across all datasets, achieving accuracy improvements of up to 29.45%.

## introduction generative ai models, including large language models (llms), are renowned for their ability to generate text that resembles human language. however, these models frequently fabricate information, a phenomenon known as hallucination @xcite . this characteristic presents both opportunities and challenges. on the one hand, hallucinations fuel creative potential; on the other, they blur the boundary between truth and fiction, introducing inaccuracies into seemingly factual statements @xcite . hallucinations in llms can generally be categorised into two main types: factual and relevance hallucinations @xcite . factual hallucinations emerge when models address topics beyond their training data, while relevance hallucinations involve factually correct content that is contextually irrelevant @xcite . * equal contribution. figure 1 : illustrative example of safe in action. the sample question is taken from the truthfulqa @xcite dataset, and the response is generated by gemma-2-9b @xcite . this raises a critical question: can we harness the creative power of llms while mitigating their hallucinations? mitigation strategies fall into two primary categories: (1) data-driven methods, which filter pre-training data or leverage highquality instruction-tuning datasets @xcite , and (2) input-side techniques, such as retrieval-augmented generation (rag), which augment queries with external, verifiable information @xcite . however, most existing approaches overlook the internal mechanisms of llms, leaving the root causes of hallucinations largely unaddressed @xcite . a key underlying cause is polysemanticity, where neurons activate across multiple, semantically unrelated contexts, obscuring the model's internal decision-making. this phenomenon often stems from superposition @xcite . recent work @xcite has introduced saes to mitigate this challenge by decomposing polysemantic activations into a large-scale dictionary of interpretable, monosemantic features. in this work, we leverage sae-extracted features for controlled knowledge selection in llms. although ai hallucinations are intrinsic to how llms function, making their complete elimination impossible (banerjee et al., 2024), we propose safe (sparse autoencoderbased framework for robust query enrichment). this method addresses this challenge using saebased feedback. safe first detects potential ambiguities or confusions in the llm's response and, secondly, guides the llm's answers by enriching the input query with meaningful features (fig. 1 presents a toy example illustrating this phenomenon). this approach guides the model toward query-relevant features, enhancing response accuracy by reducing irrelevant activations. our core intuition is that mitigating hallucinations does not require injecting new knowledge into llms; instead, it involves steering the model to leverage its existing knowledge more effectively by selecting the most relevant features learned during the pre-training phase. contributions our contributions are three-fold: the remainder of this paper is structured as follows: section 2 presents a review of related work. section 3 describes the safe pipeline in detail. the experimental setup is outlined in section 4, followed by the presentation of validation results and main experimental findings in section 5. section 6 provides an ablation study, while section 7 discusses the implications of our findings. finally, the paper concludes with an overview of the key advantages and limitations of safe in sections 8 and 9, respectively. ## methodology the safe pipeline integrates a hallucination detection framework with an sae-based approach to effectively mitigate hallucinations in llms through query enrichment. the process is depicted in fig. 2 . ## experimental setting models employed. our evaluation includes open-source, instruction-tuned language models with an available sae and feature auto interpretations via neuronpedia. specifically, we assess meta's llama 3 (8b) @xcite and gemma 2 (9b) @xcite . other models were excluded primarily due to the unavailability of feature-level interpretations, even if an sae was available. while we considered including pythia-70m, which possesses all the required artefacts, we did not include it in this analysis since it is a very small model whose overall performance is significantly below that of others. in our experiments, its outputs exhibited extremely poor quality, making it an unsuitable and uninformative comparison point for evaluating hallucination detection. ## results we adopt the value ϕ = 0.6 when using self-checkgpt and sindex, and ϕ = 0 when using halocheck for the hallucination detection portion of the pipeline, and δ = 0.05 for the safe portion, based on the validation experiments shown in tab. 2. these thresholds were chosen because self-checkgpt and sindex use a similar scale for computing 'contradiction' between generations, making ϕ = 0.6 a practical cutoff for identifying likely hallucinations. a threshold of ϕ = 0 is appropriate for halocheck, which outputs a score between -1 and 1, with scores below 0 indicating contradictions. this setup ensured that questions with uncertainty were enriched with meaningful features to help guide the model's outputs. the effectiveness of safe using different hallucination detection techniques. tab. 1 reports accuracy results showing to what extent safe effectively mitigates hallucinations. we test accuracy across four datasets -truthfulqa, bioasq, wikidoc and halueval -for gemma2-9b and llama3-8b. as the hallucination mitigation stage of safe can work with any detection algorithm, we integrated with three hallucination detection frameworks -sindex @xcite , halocheck @xcite , and selfcheck-gpt @xcite . results for gemma2-9b show an improvement of 2.80%, 3.04%, 0.81%, and 1.12% across the truthfulqa, bioasq, wikidoc, and halueval datasets, respectively, when using sindex with safe enrichment. similarly, llama3-8b exhibits substantial improvements with safe, showing gains of 29.45%, 9.84%, and 3.77% on truthfulqa, bioasq, and wikidoc, respectively, along with a 3.65% improvement on halueval. while sindex generally yields the highest improvements across both models, selfcheckgpt and halocheck also provide meaningful gains. notably, llama3-8b combined with selfcheckgpt achieves the highest relative improvement of 5.67% on halueval. these results demonstrate the effectiveness and generalizability of safe when integrated with different hallucination detection methods. ## ablation studies: component-wise performance analysis to rigorously evaluate the contribution of each component in safe, we conducted a series of ablation studies by selectively removing or modifying key elements of safe. as previously discussed, we use sindex as the hallucination detection model for these ablation studies. the results, summarized in tab. 5, provide insight into the relative importance of these components. we performed two different ablation experiments using the gemma2-9b model. ## discussion the results demonstrate the effectiveness of safe in mitigating hallucinations and improving llm performance. as shown in together, these findings demonstrate that safe's synergy of hallucination detection and sae-guided enrichment enhances the reliability of llms without requiring additional model training. ## conclusion hallucination remains a persistent challenge in llm-based applications, undermining their reliability and trustworthiness in real-world deployments. in this work, we propose safe, a sparse autoencoder-based framework for robust query enrichment, which mitigates hallucinations by refining input queries and guiding model responses through interpretable, semantically grounded feature selection. safe employs a two-stage process: first, it detects hallucinations using sota hallucination detection algorithms; then, it mitigates these issues by enriching queries with features derived from a sae. empirical evaluations across diverse benchmark datasets demonstrate that safe significantly reduces hallucination rates while improving response accuracy by up to 29.45%. ablation studies confirm the critical role of detection and sae-driven enrichment in achieving these gains. notably, safe operates in a training-free manner, offering a lightweight, plug-and-play solution that seamlessly integrates into existing llm pipelines without additional model fine-tuning.