SAC ^3 : Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency (2023)

Abstract: Hallucination detection is a critical step toward understanding the trustworthiness of modern language models (LMs). To achieve this goal, we re-examine existing detection approaches based on the self-consistency of LMs and uncover two types of hallucinations resulting from 1) question-level and 2) model-level, which cannot be effectively identified through self-consistency check alone. Building upon this discovery, we propose a novel sampling-based method, i.e., semantic-aware cross-check consistency (SAC ^3 ) that expands on the principle of self-consistency checking. Our SAC ^3 approach incorporates additional mechanisms to detect both question-level and model-level hallucinations by leveraging advances including semantically equivalent question perturbation and cross-model response consistency checking. Through extensive and systematic empirical analysis, we demonstrate that SAC ^3 outperforms the state of the art in detecting both non-factual and factual statements across multiple question-answering and open-domain generation benchmarks.

## introduction large-scale pre-trained language models (lms) have demonstrated exceptional adaptability across a diverse array of natural language tasks that require generating open-ended responses based on user prompt comprehension @xcite . however, prominent lms like gpt @xcite and palm @xcite , often exhibit a tendency to produce exceedingly confident, yet erroneous, assertions commonly referred to as hallucinations. this phenomenon significantly impedes their applicability in domains where factual accuracy is of utmost importance. hallucinations can be detected through the assistance of metrics that capture the uncertainty about 1 all resources are available at https://github.com/ intuit/sac3 . the output sequences. however, these metrics require access to token-level log probabilities, which are not available in commercial black-box lms like chatgpt or bard that only offer limited api access. to overcome this limitation, recent studies explore sampling-based approaches for approximating uncertainty estimation @xcite through establishing a connection between confidence and self-consistency @xcite . the underlying premise of this principle is that lms are more inclined to generate consistent responses when high probabilities are assigned to tokens in the answer, which, in turn, implies a level of factuality. in contrast, inconsistent responses are more likely to contain hallucinations. to operationalize this concept, current approaches are designed to sample multiple responses from the lms for a given question and then compose a hallucination score for each sentence based on the level of response consistency. ## self-consistency limitations in factuality assessment the essential assumption of self-consistency in factuality assessment is that if the lm has the knowledge of the concept, responses sampled from its output distribution, should be similar and consistent; conversely, if the lm lacks corresponding knowledge, the sampled responses would contain hallucinated facts that are diverged and contradictory. although this assumption may seem reasonable, it does not always hold in practice (more details are provided in the appendix a). specifically, we argue that solely checking the lm's selfconsistency is insufficient for detecting hallucination or verifying factuality under the following two circumstances: we observe that for certain questions, lms may output consistently wrong answers. for instance, as shown in fig. 1 , when prompted with the question "is pi smaller than 3.2?", chatgpt consistently generates incorrect answers. in this case, where the generated responses are consistent but non-factual, solely relying on self-consistency checking of a single model would yield false negative hallucination detection results. ## sac 3 : semantic-aware cross-check consistency this section describes the proposed semantic-aware cross-check consistency approach, a high-level overview of which is provided in fig. 1 . additionally, an illustrative example of each component is presented in ## data and annotation we evaluate our hallucination detection approach on two categories of qa tasks, namely, classification qa and generation qa, with each category containing two datasets. following prior work @xcite , we use the following two binary classification datasets for evaluation on the classification qa task: as for the generation qa tasks, we take questions from the following two open-domain qa datasets and generate answers using llms. then we manually annotate the factuality of the answers following previous work @xcite . we use the same setting as hotpotqa-halu to create a small-scale dataset that consists of 250 non-factual and factual examples with manual annotations. please find more relevant details about data annotations in the appendix c.3. 6 experiments 6.1 experimental setup evaluation models. we use gpt-3.5-turbo from openai as the target lm for our experiment. the verifier lm is chosen from the following two models: (1) falcon-7b-instruct (almazrouei et al., 2023): an open-source causal decoder-only model built by tii that is trained on 1,500b tokens of refinedweb @xcite and further enhanced using the curated corpora; and (2) guanaco-33b: an open-source instructionfollowing models through qlora @xcite tuning of llama @xcite base model on the oasst1 dataset. implementation details. the evaluation is conducted using azure openai api. when performing semantic perturbations and consistency checking, we set the temperature to 0.0 to get deterministic high-quality outputs. given a specific input query, we generate @xmath0.1. for the self-checking-based method sc 2 , we follow prior work @xcite to set the temperature to 1.0 and generate n @xmath1. for sac 3 -q and sac 3 -qm , we set n @xmath2. to further reduce the inference cost, we set n @xmath3. we use hallucination detection accuracy and area under the roc curve (auroc) to evaluate the performance. in addition to the estimated hallucination score, we also show the verbalized probabilities @xcite ) from the target lm for comparison. we execute all experiments on 8 nvidia v100 32g gpus. ## conclusion and discussion we investigate the relationship between the selfconsistency of lm and the factuality of the response and propose sac 3 as a robust hallucination detection approach for black-box lms. through extensive empirical analysis, our work highlights several findings. first, self-consistency checking alone is insufficient to effectively detect questionlevel and model-level hallucinations, where lms generate consistently wrong responses to certain questions. second, cross-checking between semantically equivalent questions can reduce the occurrence of persistent hallucinations, potentially by reducing question-level ambiguity. third, there exists a model-level disparity in hallucinations, which we attribute to the inherent differences in lm capabilities originating from different training procedures and data. thus, the verifier lm can be selected according to specific tasks to maximize detection accuracy. we believe that our work is an important step towards building reliable llms.