The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models (2024)

Abstract: In the era of large language models (LLMs), hallucination (the tendency to generate factually incorrect content) poses great challenges to trustworthy and reliable deployment of LLMs in real-world applications. To tackle the hallucination, three key questions should be well studied: how to detect hallucinations (detection), why do LLMs hallucinate (source), and what can be done to mitigate them (mitigation). To address these challenges, this work presents a systematic empirical study on LLM hallucinations, focused on the three aspects of hallucination detection, source and mitigation. Specially, we construct a new hallucination benchmark HaluEval 2.0, and design a simple yet effective detection method for LLM hallucinations. Furthermore, we zoom into the different training or utilization stages of LLMs and extensively analyze the potential factors that lead to the LLM hallucinations. Finally, we implement and examine a series of widely used techniques to mitigate the hallucinations in LLMs. Our work has led to several important findings to understand the hallucination origin and mitigate the hallucinations in LLMs.

## introduction large language models (llms) @xcite have shown remarkable potential in a wide range of natural language processing applications @xcite @xcite . however, despite the significant improvement in model capacity, a persistent challenge lies in their tendency to hallucinate, i.e., generate content that looks plausible but is factually incorrect @xcite @xcite . this issue severely restricts the deployment of llms in real-world applications (e.g., clinical diag- * corresponding author noses), where the reliable generation of trustworthy text is of utmost importance. in the era of llms, there has been a significant surge of research interests in hallucinations @xcite @xcite @xcite . these studies are mainly centered around three interleaved questions, i.e., why do llms hallucinate, how to detect hallucinations, and what can be done to mitigate them? the three key questions pose great challenges to the research community, while existing empirical work mostly focuses on analyzing or addressing individual challenges, still lacking a systematic and in-depth experimental study on llm hallucinations. to bridge this gap, a more comprehensive analysis is needed to thoroughly research the aforementioned three questions. for deciphering the mystery of hallucination in llms, we aim to conduct a comprehensive and systematic empirical study on hallucination detection, source, and mitigation. in particular, we mainly focus on studying factuality hallucination, which has become one of the primary erroneous sources for llms @xcite . to carry out our research, we zoom into the different stages to train and use llms, including pretraining, supervised fine-tuning (sft), reinforcement learning from human feedback (rlhf), and inference, and thus can conduct more in-depth analysis of potential impact of each stage on model hallucination. this analysis approach is quite different from prior work, where they mostly study the impact of individual stages or strategies to attribute or mitigate the hallucinations. to conduct our empirical study, we first extend previous work @xcite and construct a new benchmark halueval 2.0 for evaluating the factuality hallucination in llms. our benchmark contains 8,770 questions from five domains including biomedicine, finance, science, education, and open domain. to detect factual errors in llm responses, we propose a simple yet effective framework that decomposes the task of hallucination detection into two simple sub-tasks, i.e., extract factual statements from responses and judge the trustfulness of each statement. then, based on the detection approach, we perform a series of studies to explore the hallucination sources from four aspects, i.e., pre-training, sft, and inference methods. for each aspect, we extensively examine the effect of possible factors on llm hallucinations. finally, we delve into hallucination mitigation for llms through a series of widely used techniques, including rlhf, retrieval augmentation, and advanced decoding, etc. it is generally considered to be challenging to deeply understand and fully mitigate llm hallucinations. by providing more comprehensive empirical analysis, our work aims to further push forward the research on hallucinations. the major findings of our empirical study are summarized below: ## hallucination in llms in the field of nlp, hallucination typically refers that the model output contains undesired content that is nonsensical or deviates from the source material @xcite . despite that hallucination can be defined in different ways, we focus on factuality hallucination, since it has become one of the primary sources of erroneous responses by llms. in light of this research, we propose a fine-grained categorization of factuality hallucination in llms as follows: note that it is difficult to encompass all kinds of hallucination, and our taxonomy aims to depict the most frequently occurring types of hallucination. we present several illustrative examples for each type of hallucinations in ## experimental setup in this section, we first introduce our benchmark and then describe a set of models for comparison. ## hallucination detection to analyze and mitigate hallucinations, the first and fundamental step is to detect hallucinations. in this section, we design an automatic hallucination detection approach and further validate its reliability by comparing with human labeling. ## hallucination source in this section, we perform a series of experiments to explore four factors that may induce llm hallucinations, including pre-training, sft, inference, and prompt design. we present some experiments of hallucination source in appendix c. ## hallucination mitigation to alleviate the hallucinating behaviors of llms, several hallucination mitigation techniques have been proposed. in this part, we will study the effectiveness of three methods, including rlhf, retrieval augmentation, and advanced decoding. we also examine the effectiveness of self-reflexion and prompt improvement in appendix d. ## related work hallucination has been a fundamental challenge in llms, receiving extensive attention in existing literature @xcite @xcite . we discuss them in two aspects: source/detection and mitigation. hallucination source and detection. to understand and detect the hallucination in llms, several studies focus on using the llm itself as the tool to study the hallucinated content. for llms with access to their internal states, we can delve into the inner workings of the model to explore the principles behind hallucinations @xcite @xcite . typically, the internal states that can be studied by examining the output logit values, the hidden layer activations, and the attention states. for example, @xcite leveraged the output logit values of the model as a signal of hallucinations to estimate the uncertainty of responses. for models that can only be accessed through api calls, hallucinations are typically studied by analyzing the relationship between input prompts and the model's output responses @xcite @xcite . for example, rawte et al. (2023b) explored the effects of linguistic elements in prompts (readability, formality, and concreteness) on the llm hallucinations. ## conclusion this paper presented a comprehensive empirical analysis about llm hallucinations in the three aspects of detection, source and mitigation. we constructed the hallucination benchmark halueval 2.0 and developed an llm-based automatic detection approach. based on this benchmark, we further systematically investigated the possible sources for llm hallucination in the stages of pre-training, sft, rlhf, and inference, and also examined the effectiveness of a series of hallucination mitigation strategies, including rlhf, retrieval augmentation, self-reflexion, advanced decoding, and prompt improvement. as the major contribution, our benchmark can be reused for further research, and our work has led to a series of important empirical findings on the source and mitigation of hallucination. despite the great efforts that we have made, our analysis about pre-training stage and sft is still limited, due to the lack of disclosed training details and the supporting computational sources. we will investigate into the two stages with more detailed analysis as future work. furthermore, our experimental tests are not yet sufficient and we are also particularly interested in the working mechanism or nature of llms in generating the hallucinations. we will conduct more in-depth research work in the future. in addition, this paper mainly aims to provide empirical analysis on existing techniques to mitigate the llm hallucinations, and there are no new hallucination mitigation strategies proposed. we will also consider developing improved mitigation strategies based on the findings of this work. our hallucination detection approach is based on the gpt model (i.e., gpt-4), which might inevitably lead to some minor errors. we will continuously improve our method in the future work. and 4-bit quantization results in an overall higher level of hallucination compared to the original 16bit model. in most cases, 8-bit quantization has a minimal impact on the hallucination of the model, while 4-bit quantization significantly increases the hallucination in the model's responses. in certain domains such as biomedicine, the hallucination rate of the quantized model will noticeably increases compared to the original model. token-by-token generation. most llms adopt the token-by-token manner to generate a token at a time. once the generated part contains erroneous or unreasonable content, the model is difficult to complete the sentence correctly @xcite . to explore the drawback of token-by-token generation, we conduct a case study to qualitatively analyze how the generation paradigm leads to hallucinations. we present several examples across five domains in