Factuality of Large Language Models: A Survey (2024)

Abstract: Large language models (LLMs), especially when instruction-tuned for chat, have become part of our daily lives, freeing people from the process of searching, extracting, and integrating information from multiple sources by offering a straightforward answer to a variety of questions in a single place. Unfortunately, in many cases, LLM responses are factually incorrect, which limits their applicability in real-world scenarios. As a result, research on evaluating and improving the factuality of LLMs has attracted a lot of research attention recently. In this survey, we critically analyze existing work with the aim to identify the major challenges and their associated causes, pointing out to potential solutions for improving the factuality of LLMs, and analyzing the obstacles to automated factuality evaluation for open-ended text generation. We further offer an outlook on where future research should go.

## introduction large language models (llms) have become an integral part of our daily lives. when instructiontuned for chat, they have enabled digital assistants that can free people from the need to search, extract, and integrate information from multiple sources by offering straightforward answers in a single chat. while people naturally expect llms to always present reliable information that is consistent with real-world knowledge, llms tend to fabricate ungrounded statements, resulting in misinformation @xcite , which limits their utility. thus, assessing and improving the factuality of the text generated by llms has become an emerging and crucial research area, aiming to identify potential errors and to advance the development of more reliable llms @xcite . to this end, researchers have collected multiple datasets, introduced a variety of measures to evaluate the factuality of llms, and proposed numerous strategies leveraging external knowledge through retrieval, self-reflection, and early refinement in model generation to mitigate factual errors @xcite . numerous surveys @xcite @xcite have explored factuality or hallucinations in large language models across various modalities. while they either lack in-depth discussion or are too specific to grasp the fundamental challenges, promising solutions in factuality evaluation and enhancement, and some ambiguous concepts in llm factuality. we summarized these surveys in our survey aims to bridge this gap by providing an in-depth analysis of llm factuality, with an emphasis on recent studies to reflect the rapidly evolving nature of the field. we offer a comprehensive overview of different categorizations, evaluation methods, and mitigation techniques for llm factuality in both language and vision modalities. additionally, we explore a novel research avenue that seeks to improve llm calibration. this includes making models aware of their knowledge limitations and enhancing the reliability of their output confidence. ## background hallucination and factuality, while conceptually distinct, often occur in similar contexts and are sometimes used interchangeably, rendering them intricately intertwined, posing a challenge in discerning their distinct boundaries, and causing a considerable amount of misconception. in this section, we seek to disambiguate and refine our understanding of these two closely aligned concepts, thereby preventing misinterpretation and reducing potential confusion. additionally, we further include two closely-related axes: relevance and trustworthiness for llm evaluation to illustrate their nuance in relation to factuality. ## evaluating factuality evaluating llm factuality on open-ended generations presents a non-trivial challenge, discerning the degree to which a generated textual statement aligns with objective reality. studies employ vari- type dataset topic size er% evaluation and metrics used in original paper freq i factscore-bio (min and et al., 2023) biography 549 42.6 human annotation and automated fact-checkers 4 factcheck-gpt @xcite open-ended questions 94 64.9 human annotation 1 factool-qa @xcite knowledge-based qa 50 54.0 human annotation and automated fact-checkers 2 felm-wk @xcite knowledge-based qa 184 46.2 human annotation, accuracy and f1 score 1 halueval @xcite open-ended questions 5000 12.3 human annotation, auroc + llm judge + parent 3 freshqa @xcite open-ended questions 499 68.0 human annotation 2 selfaware @xcite open-ended questions 3369 -evaluate the llm awareness of unknown by f1-score 1 ii snowball @xcite yes/no question 1500 9.4 exact match + accuracy/f1-score 1 iii wiki-category list @xcite name some [mexican films] 55 -precision/recall@5 1 multispan qa @xcite short-term answer 428 -exact match + f1 score 1 iv truthfulqa @xcite false belief or misconception 817 -accuracy 5 hotpotqa @xcite multi-step reasoning 113k -exact match + f1 score 11 strategyqa @xcite multi-step reasoning 2780 -recall@10 3 mmlu @xcite knowledge 15700 -accuracy 4 ## improving factuality improving the factuality of an llm often requires updating its internal knowledge, editing fake, outdated and biased elements, thereby making its output reflect a revised collection of facts, maximizing the probability of p (truth|prompt). one option is to adopt gradient-based methods to update model parameters to encourage desired model output. this includes pre-training, supervised finetuning and rlxf. we can also explore injecting a new fact into llms or overwriting the false knowledge stored in llm memory by in-context learning (icl). when models store factually correct knowledge but produce errors, they can in some cases rectify them through self-reasoning, reflection, and multi-agent debates. we discuss these methods throughout the lifecycle of an llm, ranging from pre-training, to inference, to post-processing. another important element is retrieval augmentation, which enhances the generation capabilities of llms by anchoring them in external knowledge that may not be stored or contradict the information in llm parametric memory. it can be incorporated at various stages throughout model training and the subsequent inference process @xcite , and is therefore not discussed individually. ## factuality of multimodal llms factuality or hallucination in multimodal large language models refers to the phenomenon of generated responses being inconsistent with the image content. current research on multimodal factuality can be further categorized into three types: evaluation chair @xcite is the first benchmark for assessing the accuracy of object existence within captions, focusing on a predefined set of objects in the coco dataset @xcite . however, this approach can be misleading since the coco dataset is frequently used in training sets, providing a limited perspective when used as the sole basis for evaluation. in contrast, pope @xcite evaluates object hallucination with multiple binary choice prompts, both positive and negative, querying if a specific object exists in the image. more recently, @xcite proposed gpt4-assisted visual instruction evaluation (gavie) to evaluate the visual hallucination additionally, @xcite demonstrated the use of human evaluation to avoid inaccuracies and systematic biases. ## challenges and future directions we first identify three major challenges for improving the factuality of llms, and then we point to several promising directions for future work. language models learn a language distribution, not facts. the training objective of language modeling is to maximize the probability of a sentence, as opposed to that of a factual statement. while capable of generating seemingly coherent and fluent outputs upon convergence, models are not guaranteed to always return a factual response. ## conclusion we presented an overview on the factuality of llms, surveying a number of studies covering topics such as evaluation and improvement methods (applicable at various stages: pre-training, sft, inference and post-processing) along with their respective challenges. we also identified three major issues and pointed out to promising future research directions.