Self-Alignment for Factuality: Mitigating Hallucinations in LLM s via Self-Evaluation (2024)

Abstract: Despite showing impressive abilities, large language models (LLMs) often struggle with factual inaccuracies, i.e., ”hallucinations”, even when they hold relevant knowledge. To mitigate these hallucinations, current approaches typically necessitate high-quality human factuality annotations. In this work, we explore Self-Alignment for Factuality, where we leverage the self-evaluation capability of an LLM to provide training signals that steer the model towards factuality. Specifically, we incorporate Self-Eval, a self-evaluation component, to prompt an LLM to validate the factuality of its own generated responses solely based on its internal knowledge. Additionally, we design Self-Knowledge Tuning (SK-Tuning) to augment the LLM’s self-evaluation ability by improving the model’s confidence estimation and calibration. We then utilize these self-annotated responses to fine-tune the model via Direct Preference Optimization algorithm. We show that the proposed self-alignment approach substantially enhances factual accuracy over Llama family models across three key knowledge-intensive tasks on TruthfulQA and BioGEN.

## introduction despite exhibiting remarkable proficiency in a diverse range of nlp tasks @xcite @xcite , llms @xcite @xcite occasionally generate seemingly plausible yet factually incorrect statements, i.e., "hallucinations" @xcite @xcite . such hallucinations can undermine the trustworthiness and practical applicability of llms in real-world scenarios, particularly when employed in high-stakes tasks @xcite . given a prompt to write a biography, before factuality alignment, the llm generates some facts that are not accurate. through self-evaluation, the llm is capable of identifying these inaccurate facts. the feedback from the self-evaluation is used as a reward signal to align the llm towards factuality. each fact is highlighted in distinct colors, and the corrected facts are marked with green letters. in this paper, we focus on mitigating a noteworthy type of hallucination, where an llm holds relevant knowledge in response to a query (i.e., "knows"), yet occasionally falters in conveying accurate information (i.e., "tells") @xcite . for instance, an llm might generate an inaccurate response during one inference time but can provide a correct response at another time @xcite @xcite . this gap between "knowing" and "telling" @xcite @xcite significantly undermines the potential of llms to accurately convey the knowledge acquired during the pre-training phase. a few studies @xcite @xcite edit the model's internal representations towards "factuality" directions, using domain-specific annotated data. meanwhile, acknowledging the inadequacy of the training objective-maximum likelihood estimation (mle)-in accurately capturing factuality @xcite @xcite , a recent study @xcite introduces the llm's internal factuality signals as training rewards to guide the models towards factuality. given that the origin of a llm's hallucinations is intrinsically linked to its confidence foot_0 @xcite , @xcite employs consistency-based confidence regarding the factual correctness over the generate responses @xcite as the factuality signals. nevertheless, such consistencybased confidence remains rely on the model's generation ability, which might be non-reflective on model's internal knowledge. despite the challenges faced by an llm in directly "telling" the correct response, it has showed potential in "evaluating" its generated responses @xcite . as depicted in figure 1 , the llm is capable of identifying factual inaccuracies within the responses it generates, with a reasonable prediction confidence. such self-evaluation, i.e., directly prompting the model itself about internal knowledge awareness, might be a more effective approach to factuality estimation. in this paper, we introduce a self-alignment framework, self-alignment for factuality, which harnesses an llm's self-evaluation capability to mitigate hallucinations. our approach encourages an llm to generate prediction confidence scores pertaining to the factuality of its own generated responses through self-asking. subsequently, these scores are utilized as reward signals to fine-tune the model using the direct preference optimization (dpo) algorithm @xcite . specifically, we incorporate a factuality self-evaluation component, self-eval, which prompts the llm to directly validate its responses based on its internal knowledge. to bolster the llm's universal selfevaluation ability, we introduce sk-tuning to enhance the llm's internal knowledge awareness, i.e., prediction confidence estimation and calibration foot_1 @xcite , through sufficient tuning across heterogeneous knowledge-oriented tasks. we assess the effectiveness of the proposed self-alignment for factuality framework on three crucial knowledge-extensive tasks for llms, namely multi-choice question-answering (mcqa), shortform open-ended generation, and long-form openended generation, using two benchmark datasets: truthfulqa @xcite and biogen @xcite . the results show that, solely relying on the model's internal knowledge, self-alignment for factuality significantly enhances the factual accuracy of llama family models @xcite ) across all three tasks, notably surpassing the representation-editing methods @xcite and the recent work with consistency-based confidence @xcite . in summary, our contributions are three-fold: ## related work hallucinations in llms. hallucinations in llms occur when generated content, is seemingly plausible, however deviates from actual world knowledge @xcite @xcite . in this study, we align with the perspective that an llm's acquired knowledge should mirror established facts @xcite . we focus on a specific type of "unfaithful hallucination" where llms produce factually incorrect statements, even when possessing relevant knowledge @xcite @xcite . rather than broadly targeting the enhancement of llms' factuality @xcite @xcite @xcite @xcite , our goal is to align llms to reliably convey accurate information when they have sufficient knowledge. our research parallels two significant studies in the field by @xcite and @xcite . @xcite focus on honestybased fine-tuning, empowering llms to admit limitations by acknowledging "i don't know", our self-alignment for factuality approach is distinctively geared towards guiding llms to articulate truthful information when they hold pertinent knowledge. in contrast to @xcite , which relies on a consistency-based method for confidence estimation, our work introduces self-eval-skt, which is trained on a broad spectrum of heterogeneous data, and designed to enhance confidence estimation capabilities significantly. experimental results from our study demonstrate a notable improvement in the accuracy and reliability of factual information presented by llms. we provide a brief summary in appendix a. dpo in section 3.3. ## experiments in this section, we evaluate the efficacy of our proposed framework across three knowledge-intensive tasks: mcqa, short-form and long-form openended generation, following @xcite . ## conclusion in this paper, we introduce self-alignment for factuality, a framework that capitalizes on an llm's self-evaluation ability to mitigate hallucinations. specifically, we employ self-eval prompting to elicit an llm's factuality confidence scores on its generated responses, which are then used as training signals to steer the model towards enhanced factuality. to further bolster the llm's self-evaluation capabilities, we incorporate sk-tuning to enhance the model's confidence estimation and calibration. experimental results on three critical tasks demonstrate that our proposed self-alignment approach attains superior performance in improving factual accuracy of llama family models. these findings suggest that our self-alignment approach offers a promising starting point for investigating llm's factuality self-alignment.