Knowledge Verification to Nip Hallucination in the Bud (2024)

Abstract: While large language models (LLMs) have demonstrated exceptional performance across various tasks following human alignment, they may still generate responses that sound plausible but contradict factual knowledge, a phenomenon known as hallucination. In this paper, we demonstrate the feasibility of mitigating hallucinations by verifying and minimizing the inconsistency between external knowledge present in the alignment data and the intrinsic knowledge embedded within foundation LLMs. Specifically, we propose a novel approach called Knowledge Consistent Alignment (KCA), which employs a well-aligned LLM to automatically formulate assessments based on external knowledge to evaluate the knowledge boundaries of foundation LLMs. To address knowledge inconsistencies in the alignment data, KCA implements several specific strategies to deal with these data instances. We demonstrate the superior efficacy of KCA in reducing hallucinations across six benchmarks, utilizing foundation LLMs of varying backbones and scales. This confirms the effectiveness of mitigating hallucinations by reducing knowledge inconsistency. Our code, model weights, and data are openly accessible at https://github.com/fanqiwan/KCA .

## introduction fine-tuning foundation large language models (llms) in line with human preferences, such as helpfulness, harmlessness, and truthfulness @xcite , have demonstrated remarkable effectiveness. recent research indicates that these foundation llms primarily acquire their knowledge from the pretraining corpus, while the alignment process serves to facilitate the incorporation of preencoded knowledge for generating human-aligned responses rather than introducing additional knowledge @xcite . however, it should be noted that there may be certain knowledge contained in the alignment data only, but not in the pretraining corpus. as illustrated in figure 1 , the knowledge snippet regarding "direct preference optimization", a recently introduced technique for llm alignment @xcite , is absent from the pretraining corpus of the foundation llms, leading to a phenomenon termed as knowledge inconsistency between the external knowledge in alignment data and the intrinsic knowledge embedded within foundation llms. knowledge inconsistency could lead the finetuned llms to adapt to external knowledge they encounter but do not fully comprehend, resulting in hallucinated responses. our preliminary analysis in figure 2 reveals a direct correlation between the percentages of knowledge inconsistency and the rates of hallucination across various foundation llms and benchmarks. 1 therefore, an interesting question arises: how to identify alignment data instances that trigger knowledge inconsistency before conducting alignment to mitigate hallucinations. recent studies attempt to identify inconsistent instances by evaluating if foundation llms can generate correct responses for alignment questions. then, they calibrate the inconsistency by appending a sentence to the ground truth answer that describes the uncertainty of their capabilities during alignment @xcite . however, these approaches suffer from two major limitations. firstly, they are designed for tasks like question answering, which can be evaluated via a binary true/false or a single accuracy score. for complex tasks, such as "describe the process of photosynthesis in detail.", it becomes challenging to ascertain the confidence level in the gold references. this limits their applicability to simple tasks like question answering and scalability to complex realworld scenarios. secondly, these methods solely depend on instructions and responses for detecting knowledge inconsistency, neglecting the explicit integration of reference knowledge that llms require to answer the instructions. consequently, this limitation leads to difficulties in attributing and interpreting the knowledge inconsistency. in this paper, we propose a novel approach to mitigating knowledge inconsistency through knowledge consistent alignment (kca). for each alignment data instruction, kca first examines the foundation llm through a full test of the knowledge required to answer the instruction. to automatically generate testing questions, we leverage existing well-aligned llms, such as gpt-3.5, to create multiple-choice examinations related to the knowledge required to answer the instructions. then, we utilize the testing scores on the constructed questions to identify instructions with low scores as instances that require external knowledge and indicate knowledge inconsistency issues. for each instruction requiring external knowledge, we also acquire a reference knowledge snippet related to it. next, we investigate three tuning methods to adjust data involving knowledge inconsistency, including appending the knowledge snippets to the instructions, discarding the instances, and modifying the responses into refusal format, respectively. to empirically demonstrate the effectiveness of our proposed approach, we investigate the mitigation of hallucinations in instruction-tuning setups. we employ various foundation llms including pythia 7b @xcite , mistral 7b @xcite , llama-2 7b and 13b @xcite . we utilize six benchmarks ranging general instruction-following, truthful question answering, retrieval-augmented generation, and clinical report generation with both metric-based and llm-based judgments. experimental results show that kca significantly reduces knowledge inconsistency, thereby reducing hallucination rates compared to the baseline. moreover, we explore the potential impact of reducing knowledge inconsistency on other human preferences, such as helpfulness. the results reveal that most of the tuning methods within kca demonstrate comparable performance to the baseline in maintaining helpfulness. notably, among the three tuning methods, refusal tuning exhibits superior performance in mitigating hallucinations, open-book tuning excels in maintaining helpfulness, while discard tuning provides a balance between hallucination mitigation and helpfulness maintenance. to sum up, this work distinguishes itself from previous studies in two significant aspects. firstly, our method detects knowledge inconsistency by designing assessments related to the knowledge in alignment data, regardless of task types and complexities, which can be scaled to more complex realworld scenarios. secondly, our approach explicitly leverages reference knowledge to detect knowledge inconsistency, thereby facilitating the attribution of inconsistencies within alignment data and revealing the comprehension level of foundation llms across various knowledge domains. this could contribute to the development of more comprehensive and robust llms from a knowledge perspective. ## related work the challenge of knowledge hallucination in llms @xcite ) has emerged following their remarkable performance across a diverse range of nlp tasks, especially af-ter human alignment. recent research efforts have focused on addressing hallucination throughout the entire development process of llms, including training, inference, and data curation. for the training processing, @xcite propose to use topic prefixes when designing the training objective to enhance awareness of facts. fact-rlhf @xcite develops a factaugmented approach of reinforcement learning from human feedback to supplement the reward model with additional factual information. as for the inference process, dola @xcite is based on the assumption that factual knowledge is localized in specific layers of llms. it then introduces a decoding method that contrasts the output distributions obtained from the final layer versus intermediate layers. cove @xcite leverages llms' strong ability to examine responses they produce and correct mistakes. additionally, kgr @xcite introduces an external knowledge graph during the reasoning process to provide factual knowledge for refining drafts generated by llms. for data curation, @xcite and @xcite propose to identify the knowledge boundary of llms by relying on annotated question-answer pairs to evaluate whether llms can generate correct responses. they then append a sentence to the ground truth answer describing the uncertainty of their capabilities before conducting alignment. the proposed kca approach in this paper falls within the data curation category. in contrast to these methods, which are applicable only to simple tasks like question answering, our method uses a well-aligned llm to design assessments related to the knowledge in alignment data and can be applied to various tasks of different complexities. moreover, our approach explicitly leverages reference knowledge to detect knowledge inconsistency, facilitating the attribution of inconsistencies within alignment data and revealing the capabilities of foundation llms in various knowledge domains. ## knowledge consistent alignment the core idea behind the proposed knowledge consistent alignment (kca) is to first detect inconsistencies between the knowledge in alignment data and foundation llms memorized from the pretraining corpus, and then calibrate these inconsistent instances before conducting alignment. in the following sections, we first introduce the problem settings in section 3.1, and then explain the knowledge inconsistency detection in section 3.2. finally, we explore the implementation of knowledge inconsistency calibration in section 3.3. ## experiments we assess the effectiveness of our knowledge consistent alignment (kca) method in reducing hallucinations during instruction-tuning, a prevalent approach for alignment. our experiments feature various foundation llms with different architectures and scales, where we apply kca to identify and handle knowledge inconsistencies. we evaluate our method across six public benchmarks, considering both metric-based and llm-based assessments. moreover, we explore how kca impacts other model capabilities, such as helpfulness. finally, we discuss the benefits of the three calibration strategies for addressing knowledge inconsistencies and their respective application scenarios. ## conclusion in this study, we explore the inconsistency between external knowledge within the instruction-tuning data for alignment and the intrinsic knowledge that foundation llms memorized from the pretraining corpus. we demonstrate the correlation between knowledge inconsistency and hallucinations. to address this issue, we introduce knowledge consistent alignment (kca). kca formulates test examinations for the external knowledge required to answer the instructions and evaluates the llms' comprehension based on the examination scores. moreover, kca introduces several simple yet effective strategies for knowledge inconsistency calibration, including open-book tuning, discard tuning, and refusal tuning. through a series of experiments, we demonstrate that mitigating knowledge inconsistency leads to a reduction in the hallucination rate. furthermore, we demonstrate the superiority of the proposed kca across llms of different scales and backbones on six benchmarks, employing both metric-based and llm-based judgment. compared to previous studies, the proposed kca demonstrates scalability to scenarios involving diverse task types and complexities. moreover, our approach explicitly utilizes reference knowl-edge to detect knowledge inconsistency, thereby facilitating the attribution of inconsistencies within alignment data. this allows for a comprehensive understanding of the comprehension level of foundation llms across various knowledge domains.