Generate, Discriminate, Evolve: Enhancing Context Faithfulness via Fine-Grained Sentence-Level Self-Evolution (2025)

Abstract: Improving context faithfulness in large language models is essential for developing trustworthy retrieval augmented generation systems and mitigating hallucinations, especially in long-form question answering (LFQA) tasks or scenarios involving knowledge conflicts. Existing methods either intervene LLMs only at inference without addressing their inherent limitations or overlook the potential for self-improvement. In this paper, we introduce GenDiE(Generate, Discriminate, Evolve), a novel self-evolving framework that enhances context faithfulness through fine-grained sentence-level optimization. GenDiE combines both generative and discriminative training, equipping LLMs with self-generation and self-scoring capabilities to facilitate iterative self-evolution. This supports both data construction for model alignment and score-guided search during inference. Furthermore, by treating each sentence in a response as an independent optimization unit, GenDiE effectively addresses the limitations of previous approaches that optimize at the holistic answer level, which may miss unfaithful details. Experiments on ASQA (in-domain LFQA) and ConFiQA (out-of-domain counterfactual QA) datasets demonstrate that GenDiE surpasses various baselines in both faithfulness and correctness, and exhibits robust performance for domain adaptation.

## introduction large language models have achieved remarkable success across various natural language processing tasks @xcite @xcite . despite their impressive performance, llms are prone to hallucinations-generating plausible yet nonfactual information @xcite . this limitation poses critical risks in domains where accuracy and reliability are paramount. retrieval-augmented generation (rag) has emerged as a promising framework to mitigate hallucinations and enhance context-faithfulness @xcite by grounding llm outputs in provided knowledge @xcite @xcite . despite its advantages, knowledge conflicts @xcite between llms' parametric memory and external context can undermine its effectiveness. llms may overly rely on their internal priors while disregard provided contexts, failing to meet user requirements or incorporate the latest updates @xcite . additionally, when tasked with long-form generation, such as long-form question answering (lfqa) @xcite that aims to provide in-depth and paragraph-length responses, maintaining contextfaithfulness throughout the text is still challenging @xcite . consequently, it is crucial to develop robust mechanism to alleviate faithfulness hallucination and ensure trustworthiness. many recent efforts enhance context faithfulness of llms through inference-time interventions, such as improving prompting strategies @xcite and context-aware decoding @xcite to increase the output probability on contextual information. while effective, these method do not fundamentally address the models' inherent limitations @xcite . training-based approaches, including adaptive retrieval and self-critiquing generations with reflection tokens @xcite and aligning llms via direct preference optimization (dpo) towards faithful responses @xcite , improve faithfulness by updating model parameters. however, these approaches typically rely on one-round optimization and do not fully explore the potential of continuous refinement. furthermore, existing methods often train models in complete answer level holistically, which can overlook unfaithful details, particularly in the cases of longform generation @xcite . to address these challenges, we introduce a novel self-evolving framework gendie (generate, discriminate, evolve) for enhancing llms' context faithfulness through fine-grained, sentencelevel optimization. our framework addresses the limitations of conventional answer-level training paradigms by operating at the sentence level, treating each constituent sentence of a response as an independent optimization unit. this paradigm enables fine-grained control over the quality of a complete response. central to our approach is a unified training strategy that integrates sentencelevel generation and evaluation capabilities. specifically, 1) during training, the model learns to produce context-grounded sentences while developing discriminative capabilities to distinguish between faithful and unfaithful sentences. between every two iterations, gendie generates candidate sentences through tree-structured sampling, selfscores their faithfulness, and constructs contrastive sentence pairs serving as training data in the next iteration. this design enables continuous model improvement through successive self-evolve cycles. 2) for inference, we design hierarchical decoding that first generates candidate sentences through standard methods, then selects optimal outputs using the model's learned scoring capacity to fully utilize both generative and discriminative capabilities that conventional single-stage decoding neglects. overall, the sentence-level paradigm enables both fine-grained supervision for training and a comprehensive search for inference through optimization in sentence space. we evaluate gendie on two benchmarks for context-faithful generation and the results demonstrate its effectiveness. gendie surpasses various baselines in two dimensions, faithfulness and correctness, and exhibits robust performance even in out-of-domain settings. remarkably, our approach enables the model to consistently improve with each successive training iteration. we further conduct comprehensive experiments to verify the effectiveness of the self-scoring function, as well as the superiority of sentence-level optimization. in summary, we make the following contributions: ## methodology task formulation we focus on long-form question answering (lfqa) task which requires models to generate long and detailed answers by leveraging the evidence documents provided in the input @xcite . our goal is to train the model m to generate faithful long-form answer @xmath0, a 2 , ...a |a| }, where a i denotes the ith sentence, in response to a given input question q and its corresponding evidence passages @xmath1. for training, we assume the availability of answer labels a * from a seed (in-domain) dataset to construct the initial training data. to comprehensively assess context faithfulness, we also employ a counterfactual dataset for out-of-domain evaluation. overview we present gendie, a self-evolving framework ( §2.1.2) that enhances llms contextfaithfulness in fine-grained sentence-level optimization ( §2.1.1), integrating both generation and discrimination capabilities. this enables llms to distinguish between faithful and unfaithful responses, facilitating self-scoring for both training data construction ( §2.2.2) and score-guided search during inference ( §2.3). our approach employs iterative self-training, where the models can self-generate, self-score, and therefore self-improve through multiple training iterations. ## main results gendie with greedy search outperforms most of training-free and training-based baselines in various metrics. furthermore, even with the same checkpoint, our approach earns significant boost by using hierarchical inference over greedy search or vanilla beam search, demonstrating the benefit of sentence-level search during inference. bold and underline numbers denote the best and second-best performance. note that extractive sentence selection directly uses input passage sentences as answers, making its faithfulness inherently 100%. consequently, we denote its faithfulness as "-" to indicate that faithfulness evaluation is not applicable when compared to other generative approaches. see additional experimental results in app. c. for training-free methods, prompting with gpt-4o performs well on asqa dataset, achieving the best em recall, mainly due to its extensive world knowledge obtained during pretraining. however, gpt-4o faces challenges in knowledge conflict scenarios on confiqa dataset, as it heavily relies on its own knowledge rather than the provided passages. this observation aligns with the findings reported by @xcite . a similar pattern is observed in prompting with llama, implying that simply with prompting-based method, the models often disregard those external knowledge that conflicts with their parametric knowledge. extractive sentence selection selects the most relevant sentences as answers (and thereby enjoys high faithfulness), but the concatenation of separated sentences, rather than free-form answer generation, often results in low correctness and readability due to its inflexibility. focusing on training-based methods, with the same training data, gendie gold-answer outperform standard sft, especially in terms of faithfulness, demonstrating the efficacy of our multi-task training objective. this suggests that the additional training with discrimination objective can also contribute to faithful generation. furthermore, gendie with greedy search is superior to gendie gold-answer across most metrics, which can be attributed to the iterative update of training data (more discussion in §5.1). however, gendie answer-level , which also underwent data update, shows lower faithfulness than standard sft on asqa. this discrepancy between above two comparisons highlights the necessity of sentencelevel operation for the data update, as evaluating faithfulness at sentence level is more feasible than at answer level ( §5.3). ## analysis in this section, we conduct ablation studies on the key components of our approach to evaluate their impact. unless otherwise specified, vanilla greedy search is employed for decoding in all experiments within this section. details of some variants in experiments are in §3.3. ## related work faithfulness although large language models (llms) have showcased impressive capabilities in various tasks @xcite @xcite , they are often @xcite where is the rogers cup held in montreal? ## conclusion we introduce gendie, a self-evolved approach for enhancing context-faithful generation. a distinctive feature of gendie is its capability for generating-then-self-scoring, which facilitates the model's self-evolution via iterative updates to the training data. additionally, gendie functions at the sentence level, enabling fine-grained control over faithfulness. experiments on benchmarks of long-form qa and counterfactual qa show that gendie achieves superior performances in both faithfulness and correctness over various baselines. we also verify the effectiveness of the key designs of gendie. gendie demonstrates promise in building self-evolved and trustworthy rag system.