S elf C heck GPT : Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023)

Abstract: Generative Large Language Models (LLMs) such as GPT-3 are capable of generating highly fluent responses to a wide variety of user prompts. However, LLMs are known to hallucinate facts and make non-factual statements which can undermine trust in their output. Existing fact-checking approaches either require access to the output probability distribution (which may not be available for systems such as ChatGPT) or external databases that are interfaced via separate, often complex, modules. In this work, we propose “SelfCheckGPT”, a simple sampling-based approach that can be used to fact-check the responses of black-box models in a zero-resource fashion, i.e. without an external database. SelfCheckGPT leverages the simple idea that if an LLM has knowledge of a given concept, sampled responses are likely to be similar and contain consistent facts. However, for hallucinated facts, stochastically sampled responses are likely to diverge and contradict one another. We investigate this approach by using GPT-3 to generate passages about individuals from the WikiBio dataset, and manually annotate the factuality of the generated passages. We demonstrate that SelfCheckGPT can: i) detect non-factual and factual sentences; and ii) rank passages in terms of factuality. We compare our approach to several baselines and show that our approach has considerably higher AUC-PR scores in sentence-level hallucination detection and higher correlation scores in passage-level factuality assessment compared to grey-box methods.

## introduction large language models (llms) such as gpt-3 @xcite and palm @xcite are capable of generating fluent and realistic responses to a variety of user prompts. they have been used in many applications such as automatic llm e.g. gpt-3 giuseppe mariani was an italian professional footballer who played as a forward. he was born in milan, italy. he died in rome, italy. figure 1 : selfcheckgpt with prompt. each llm-generated sentence is compared against stochastically generated responses with no external database. a comparison method can be, for example, through llm prompting as shown above. a possible approach of hallucination detection is to leverage existing intrinsic uncertainty metrics to determine the parts of the output sequence that the system is least certain of @xcite . however, uncertainty metrics such as token probability or entropy require access to token-level probability distributions, information which may not be available to users for example when systems are accessed through limited external apis. an alternate approach is to leverage fact-verification approaches, where evidence is retrieved from an external database to assess the veracity of a claim @xcite . however, facts can only be assessed relative to the knowledge present in the database. addition-ally, hallucinations are observed over a wide range of tasks beyond pure fact verification @xcite . in this paper, we propose selfcheckgpt, a sampling-based approach that can detect whether responses generated by llms are hallucinated or factual. to the best of our knowledge, selfcheck-gpt is the first work to analyze model hallucination of general llm responses, and is the first zero-resource hallucination detection solution that can be applied to black-box systems. the motivating idea of selfcheckgpt is that when an llm has been trained on a given concept, the sampled responses are likely to be similar and contain consistent facts. however, for hallucinated facts, stochastically sampled responses are likely to diverge and may contradict one another. by sampling multiple responses from an llm, one can measure information consistency between the different responses and determine if statements are factual or hallucinated. since selfcheckgpt only leverages sampled responses, it has the added benefit that it can be used for black-box models, and it requires no external database. five variants of selfcheckgpt for measuring informational consistency are considered: bertscore, question-answering, n-gram, nli, and llm prompting. through analysis of annotated articles generated by gpt-3, we show that selfcheckgpt is a highly effective hallucination detection method that can even outperform greybox methods, and serves as a strong first baseline for an increasingly important problem of llms. ## grey-box factuality assessment this section will introduce methods that can be used to determine the factuality of llm responses in a zero-resource setting when one has full access to output distributions. 2 we will use 'factual' to define when statements are grounded in valid information, i.e. when hallucinations are avoided, and 'zero-resource' when no external database is used. ## black-box factuality assessment a drawback of grey-box methods is that they require output token-level probabilities. though this may seem a reasonable requirement, for massive llms only available through limited api calls, such token-level information may not be available (such as with chatgpt). therefore, we consider black-box approaches which remain applicable even when only text-based responses are available. ## selfcheckgpt selfcheckgpt is our proposed black-box zeroresource hallucination detection scheme, which operates by comparing multiple sampled responses and measuring consistency. notation: let r refer to an llm response drawn from a given user query. selfcheckgpt draws a further n stochastic llm response samples {s 1 , s 2 , ..., s n , ..., s n } using the same query, and then measures the consistency between the response and the stochastic samples. we design selfcheckgpt to predict the hallucination score of the i-th sentence, s(i), such that s(i) ∈ [0.0, 1.0], where s(i) → 0.0 if the i-th sentence is grounded in valid information and s(i) → 1.0 if the i-th sen-tence is hallucinated. 3 the following subsections will describe each of the selfcheckgpt variants. ## data and annotation as, currently, there are no standard hallucination detection datasets available, we evaluate our hallucination detection approaches by 1) generating synthetic wikipedia articles using gpt-3 on the individuals/concepts from the wikibio dataset @xcite ; 2) manually annotating the factuality of the passage at a sentence level; 3) evaluating the system's ability to detect hallucinations. wikibio is a dataset where each input contains the first paragraph (along with tabular information) of wikipedia articles of a specific concept. we rank the wikibio test set in terms of paragraph length and randomly sample 238 articles from the top 20% of longest articles (to ensure no very obscure concept is selected). gpt-3 (text-davinci-003) is then used to generate wikipedia articles on a concept, using the prompt "this is a wikipedia passage about {concept}:". we then annotate the sentences of the generated passages using the guidelines shown in figure 3 such that each sentence is classified as either: of the 1908 annotated sentences, 761 (39.9%) of the sentences were labelled major-inaccurate, 631 (33.1%) minor-inaccurate, and 516 (27.0%) accurate. 201 sentences in the dataset had annotations from two different annotators. to obtain a single label for this subset, if both annotators agree, then the agreed label is used. however, if there is disagreement, then the worse-case label is selected (e.g., {minor inaccurate, major inaccurate} is mapped to major inaccurate). the inter-annotator agreement, as measured by cohen's κ @xcite , has κ values of 0.595 and 0.748, indicating moderate and substantial agreement ( @xcite for the 3-class and 2-class scenarios, respectively. 4 furthermore, passage-level scores are obtained by averaging the sentence-level labels in each passage. the distribution of passage-level scores is shown in figure 4 , where we observe a large peak at +1.0. we refer to the points at this peak as total hallucination, which occurs when the information of the response is unrelated to the real concept and is entirely fabricated by the llm. 7 experiments the generative llm used to generate passages for our dataset is gpt-3 (text-davinci-003), the stateof-the-art system at the time of creating and annotating the dataset. to obtain the main response, we set the temperature to 0.0 and use standard beam search decoding. for the stochastically generated samples, we set the temperature to 1.0 and generate @xmath0. for the proxy llm approach, we use llama @xcite , one of the bestperforming open-source llms currently available. for selfcheckgpt-prompt, we consider both gpt-3 (which is the same llm that is used to generate passages) as well as the newly released chatgpt (gpt-3.5-turbo). more details about the systems in selfcheckgpt and results using other proxy llms can be found in the appendix. ## conclusions this paper is the first work to consider the task of hallucination detection for general large language model responses. we propose selfcheck-gpt, a zero-resource approach that is applicable to any black-box llm without the need for external resources, and demonstrate the efficacy of our method. selfcheckgpt outperforms a range of considered grey-box and black-box baseline detection methods at both the sentence and passage levels, and we further release an annotated dataset for gpt-3 hallucination detection with sentencelevel factuality labels.