Steering LVLM s via Sparse Autoencoder for Hallucination Mitigation (2025)

Abstract: Large vision-language models (LVLMs) have achieved remarkable performance on multimodal tasks. However, they still suffer from hallucinations, generating text inconsistent with visual input, posing significant risks in real-world applications. Existing approaches to address this issue focus on incorporating external knowledge bases, alignment training, or decoding strategies, all of which require substantial computational cost and time. Recent works try to explore more efficient alternatives by adjusting LVLMs’ internal representations. Although promising, these methods may cause hallucinations to be insufficiently suppressed or lead to excessive interventions that negatively affect normal semantics. In this work, we leverage sparse autoencoders (SAEs) to identify semantic directions closely associated with faithfulness or hallucination, extracting more precise and disentangled hallucination-related representations. Our analysis demonstrates that interventions along the identified faithful direction can mitigate hallucinations, while those along the hallucinatory direction can exacerbate them. Building on these insights, we propose **S**teering LVLMs via **S**AE **L**atent Directions (SSL), a plug-and-play method based on SAE-derived latent directions to mitigate hallucinations in LVLMs. Extensive experiments demonstrate that SSL significantly outperforms existing decoding approaches in mitigating hallucinations, while maintaining transferability across different model architectures with negligible additional time overhead. The code is available at [https://github.com/huazhenglin2003/SSL](https://github.com/huazhenglin2003/SSL).

## method in this work, we introduce ssl, a plug-and-play method for steering lvlms. our method consists of two principal components: semantic directions identification and steering lvlms via sae latents. ## results ## related works lvlms refer to the phenomenon where the generated textual content is inconsistent with the visual input. this issue arises from various factors, such as dataset bias, insufficient visual perception by the encoder, and misalignment across modalities @xcite unlike previous methods, our work directly identifies hallucinatory and faithful semantic directions using saes. we then dynamically adjust these directions during visual-linguistic fusion and generation to proactively reduce hallucination outputs. furthermore, our approach contributes to the practical interpretability of saes in lvlms, demonstrating their potential for understanding and controlling internal semantic representations. ## conclusion this work explores the relationship between the hallucination in lvlms and their internal latent representations. we construct a residual stream dataset for hallucinatory and faithful object tokens, and use sae to extract the semantic directions corresponding to hallucination and factuality. based on this insight, we propose ssl, a plug-and-play method that amplifies true semantics while suppressing potential hallucinations. extensive experiments demonstrate that ssl outperforms existing methods. furthermore, although the sae was trained on llava-next, the semantic directions it extracted generalize well across different model architectures, further showcasing the potential of sae in understanding and controlling the internal semantic representations of models.