On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation (2025)

Abstract: Hallucination occurs when large language models exhibit behavior that deviates from the boundaries of their knowledge during response generation. To address this critical issue, previous learning-based methods attempt to finetune models but are limited by off-policy sampling and coarse-grained feedback. In this paper, we present Reinforcement Learning for Hallucination (RLFH), an on-policy self-alignment approach that enables LLMs to actively explore their knowledge boundaries and self-correct generation behavior through fine-grained feedback signals. RLFH introduces a self-assessment framework where the policy serves as its own judge. Through this framework, responses are automatically decomposed into atomic facts and their truthfulness and informativeness are assessed against external knowledge sources. The resulting fine-grained feedback at the statement level are then converted into token-level dense reward signals. This enables online reinforcement learning to achieve precise and timely optimization without human intervention. Comprehensive evaluations on HotpotQA, SQuADv2, and Biography benchmarks validate RLFH’s effectiveness in hallucination mitigation.

## introduction large language models (llms) have demonstrated capabilities in generating fluent and plausible responses. however, these models occasionally fabricate facts in their responses, referred to as hallucination. the crux of hallucination is the misalignment between models' generation and their internal knowledge @xcite . this misalignment manifests in various ways. for instance, as shown in figure 1 , the response of llms about * corresponding authors. @xcite turing" contains erroneous factual information, such as stating that he was born in 1911 and was american. more broadly, these hallucinations can be categorized into several types: (1) misleading responses, when the model inaccurately answers questions within its knowledge boundary; (2) reckless attempts, when the model responds to queries beyond its knowledge; and (3) evasive ignorance, when the model refrains from providing answers despite possessing the knowledge. unfortunately, due to the opaque nature of model knowledge, we can only observe erroneous model responses or their refusal to respond, without accurately determining whether they have experienced hallucinations. recent studies have attempted to mitigate hallucination in large language models via learningbased and editing-based approaches. learningbased methods first detect the model's knowledge boundaries and then finetune it with carefully curated feedback data. however, these methods face several challenges. first, due to off-policy data sampling @xcite @xcite , they experience distribution shifts, resulting in suboptimal models @xcite . second, coarse-grained instancelevel feedback @xcite @xcite fails to precisely pinpoint the hallucinations, as a single response may contain both correct and incorrect facts. finally, given our limited understanding of how models learn and express knowledge, existing knowledge detection techniques @xcite @xcite may produce inconsistent results, thus failing to accurately reflect the model's knowledge boundaries. in contrast, editing-based methods @xcite first generate content and then edit it based on external knowledge. these methods face two funda-alan turing was born in 1911 and died in 1954 in new york. he is an american computer scientist and logician and the father of artificial intelligence. alan turing is a computer scientist, who was born in 1992 in london. alan turning was born in 1911 and died in 1954 in new york. he is an american computer scientist and logician and the father of artificial intelligence. alan turning was born in 1911 and died in 1954 in new york. he is an american computer scientist and logician and the father of artificial intelligence. ## conclusion in this work, we introduce an on-policy selfalignment approach that enables llms to explore their knowledge and self-correct hallucination behavior. our approach features a self-assessment framework where the policy serves as its own judge, automatically providing fine-grained feedback through atomic fact verification and generating token-level dense rewards for online reinforcement learning. comprehensive evaluations demonstrate that our approach effectively mitigates hallucination. our work represents a step towards more reliable and self-aware language models.