Language Models Hallucinate, but May Excel at Fact Verification (2024)

Abstract: Recent progress in natural language processing (NLP) owes much to remarkable advances in large language models (LLMs). Nevertheless, LLMs frequently “hallucinate,” resulting in non-factual outputs. Our carefully-designed human evaluation substantiates the serious hallucination issue, revealing that even GPT-3.5 produces factual outputs less than 25% of the time. This underscores the importance of fact verifiers in order to measure and incentivize progress. Our systematic investigation affirms that LLMs can be repurposed as effective fact verifiers with strong correlations with human judgments. Surprisingly, FLAN-T5-11B , the least factual generator in our study, performs the best as a fact verifier, even outperforming more capable LLMs like GPT3.5 and ChatGPT. Delving deeper, we analyze the reliance of these LLMs on high-quality evidence, as well as their deficiencies in robustness and generalization ability. Our study presents insights for developing trustworthy generation models.

## introduction llms have demonstrated remarkable performance across various natural language generation (nlg) tasks @xcite @xcite . however, they persistently suffer from the hallucination problem @xcite , often generating non-factual and sometimes misleading outputs. this is quantitatively substantiated by the first part of this paper. in our carefully designed human evaluation of several current llms, gpt-3.5 only manages to produce factual outputs less than 25% of the time; other models perform even worse. such underperformance is achieved on wikipedia, a domain that they have been extensively trained in and intuitively "familiar with." our findings highlight the serious challenge that the hallucination issue presents, and underscore the crucial importance of developing effective fact verification methods @xcite . these methods are central to evaluating and incentivizing progress in improving llms' factuality. in the second part of the paper, we explore the prospect of leveraging instruction-tuned llms for fact verification. we hypothesize that, despite struggling to generate factual outputs, they may still be able to judge whether a piece of text is factual-a task that intuitively appears easier, at least for sentence-level judgments. our systematic investigation affirms this hypothesis, especially when llms are augmented with retrieval components. specifically, given a statement to be verified, we retrieve evidence from an external corpus and reframe the statement and evidence into a prompt to instruct an llm to judge the factuality. we then normalize the llm's generation probabilities of pre-defined answers as the factuality score, which shows stronger correlations with human judgments than previous statistical and model-based methods. extensive experiments further reveal that flan-t5 11b @xcite , the least factual generator in our study, even surprisingly outperforms gpt3.5 and chatgpt for fact verification. we further analyze the llm-based fact verifiers from the following perspectives: (1) influence of given evidence: chatgpt is susceptible to irrelevant evidence but deals with relevant but counterfactual evidence better than flan-t5 11b . fact verification lots of datasets have been collected towards fact verification in various domains, e.g., politics @xcite , encyclopedia @xcite @xcite , news @xcite , climate @xcite , science @xcite , and healthcare @xcite . @xcite aggregated multiple datasets to assess the ability to measure input-output consistency. statements in all above datasets usually contain only single sentences and are crafted by either crawling from dedicated web-sites @xcite , manually mutating sentences from factual articles @xcite or re-framing qa pairs @xcite . we further involve model-generated statements and paragraph-level evaluation in our study. llms as evaluators there are many active efforts to use llms' generated answers or generation probabilities for nlg evaluation @xcite @xcite . more recent studies show high correlations of chatgpt with human judgments for evaluating summarization, story generation, etc. @xcite @xcite . selfcheckgpt @xcite judged the factuality of a model output based on its similarity with other sampled outputs from the same model, and does not apply to non-model-generated statements or model-agnostic generation. factscore @xcite used llms to evaluate people's biography generation through the fraction of atomic facts supported by retrieved evidence. in contrast, we focus on openended generation around various entities and analyze llms' robustness and generalization ability. ## repurposing llms as fact verifiers our second inquiry lies around the question can llms be repurposed as effective fact verifiers? we define fact verification as follows: given a statement s and its leading context c, the verifier should give a probability p of s being factual foot_4 . the context may be absent, and the statement is a sentence or paragraph. additionally, we retrieve an evidence set, denoted as @xmath0, e 2 , • • • , e m }, from external corpora @xcite with the concatenation of c and s as the query. each piece of evidence is a passage. next, we first describe the evaluation sets ( §4.1), our verification method ( §4.2), compared verifiers ( §4.3), and then present the experiment results ( §4.4). ## analysis on llm-based verifiers this section further analyzes the influence of given evidence on llm-based verifiers ( §5.1), as well as their robustness ( §5.2) and generalization ability ( §5.3). appendix c.8 also investigates how llms' memorization of inputs may influence their judgments. and appendix c.9 further shows several representative cases that llms fail to judge, to provide more insights. ## conclusion we present a comprehensive study around two research questions: the extent to which current llms hallucinate; and their potential as effective fact verifiers. firstly, we quantitatively affirm the significant hallucination issue of current llms through a well-designed human evaluation. this highlights the urgent need for powerful fact verifiers to measure and incentivize progress in llms' factuality. to this end, we repurpose the llms into fact verifiers, and examine their ability to judge the factuality of model-generated and human-crafted statements. further analysis shows their heavy reliance on high-quality evidence and discusses their robustness and generalization ability. the evaluation suite and implemented verifiers in this paper can facilitate further research on fact verifiers and trustworthy generation models.