A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation (2022)

Abstract: Large pretrained generative models like GPT-3 often suffer from hallucinating non-existent or incorrect content, which undermines their potential merits in real applications. Existing work usually attempts to detect these hallucinations based on a corresponding oracle reference at a sentence or document level. However ground-truth references may not be readily available for many free-form text generation applications, and sentence- or document-level detection may fail to provide the fine-grained signals that would prevent fallacious content in real time. As a first step to addressing these issues, we propose a novel token-level, reference-free hallucination detection task and an associated annotated dataset named HaDeS (HAllucination DEtection dataSet). To create this dataset, we first perturb a large number of text segments extracted from English language Wikipedia, and then verify these with crowd-sourced annotations. To mitigate label imbalance during annotation, we utilize an iterative model-in-loop strategy. We conduct comprehensive data analyses and create multiple baseline models.

## introduction automatic text generation using neural natural language generation (nlg) systems is increasingly fluent and thus seemingly plausible in many realworld applications. large-scale pretrained models like gpt-3 @xcite ) are proven to be powerful in understanding and performing free form text generation tasks at human-quality level with a few in-context examples, which dramatically reduces the manual labor needed in many text-based applications and services. despite their great success, however, neural nlg systems using very large pre-trained models struggle to generate factually accurate and trustworthy text @xcite , and exhibit a propensity to hallucinate non-existent or incorrect content that is unacceptable in most user-oriented applications. this poses a major challenge for deploying production nlg systems with realtime generation, where post-examination is impossible. existing work has sought to detect hallucination and quantitatively measure generation consistency against a provided reference. such reference-based hallucination detection has been proposed for abstractive summarization @xcite , machine translation @xcite , datato-text generation @xcite , and image caption generation @xcite . for many free-form text generation tasks, however, references are not readily available. for example, in a production nlg system such as a social chatbot using real-time response generation or a document auto-completion system, the generation model often cannot pair its outputs with sufficient reference information, rendering reference-based methods less applicable: i) it may be difficult to even know where to obtain the reference, as obtaining it may be as hard as generating consistent information in the first place; ii) generation may be at a real-time online setting that demands leveraging only existing context to create new content. one common setup for qualitatively measuring the level of hallucination is performed at sentenceor document-level @xcite . related tasks such as fake news detection @xcite or fact checking @xcite also adopt this strategy. however, sentence-or document-level detection may not always provide high-resolution signals sufficient to pinpoint the hallucinated text, or can only judge whether a generated sentence or a document input: …. she had a large family and lived with her grandparents …. in 1933 she gave birth to her first child …. in july 1926, many of her friends attended her funeral … label1: @xmath0: @xmath1: operation valkyrie ( german : unternehmen walkure ) was a german world war ii emergency continuity … civil order of the nation. failure of the army to assume control of civil order might have been caused by the allied bombing of german cities , or because of the millions of jewish forced laborers employed by german factories . … modified the plan with the intention of using it to take control of german forces , to directly attack the ss , and arrest the ss leaders … label: to directly @xmath2. consequently, these high-level strategies may be insufficient to avoid hallucinations. as an alternative, at decoding time of an nlg system, we suggest that if the locus of hallucination can be identified at the token level, it may be possible to guide beam search or suppress the probability of certain tokens at real-time. to this end, we propose a reference-free, tokenlevel hallucination detection task and introduce an annotated training and benchmark testing dataset that we call hades (hallucination detection dataset). the reference-free property of this task yields greater flexibility in a broad range of generation applications. we expect the token-level property of this task to foster the development of models that can detect fine-grained signals of potential hallucination. in conjunction with consulting context to identify self-contradictory statements and access to commonsense and world knowledge, such fine-grained signals, when detected, should further mitigate real-time hallucination. our contributions include: 1) we propose a reference-free, token-level hallucination detection task for free-form text generation. 2) we support this task with a dataset that we call hades, with ∼11k instances extracted from english wikipedia using an iterative data collection strategy to address data imbalance issues. we also present comprehensive analyses on the statistical features to shed light on what is commonly recognized as hallucination in crowd-sourced judgments and its salient characteristics in free-form text generation. 3) we create multiple baselines, including feature based models and pretrained models as a first step towards addressing the proposed task. ## task overview we formulate our hallucination detection task as a binary classification task. as shown in fig 1 , our goal is to assign either a "hallucina-raw text: … failure of the government 1 to maintain 2 control of civil affairs 3 might have been caused by the allied bombing of german cities , or uprising 4 of the millions of foreign 5 forced laborers working in 6 german factories . … modified the plan with the intention of using it to take control of german cities 10 , disarm 11 the ss , and arrest the nazi leadership 12 … perturbed text: … failure of the army 1 to assume 2 control of civil order 3 might have been caused by the allied bombing of german cities , or because 4 of the millions of jewish 5 forced laborers employed by 6 german factories . … modified the plan with the intention of using it to take control of german forces 10 , to directly attack 11 the ss , and arrest the ss leaders 12 … do you think 'disarm 11 ' in src consistent waith 'to directly attack 11 ' in sub according to the context? (a) contextual perturbation (b) human annotation not really. i think 'to directly attack 11 ' in sub is inconsistent with'disarm 11 ' in src. do you think 'working in 6 ' in src consistent with 'employed by 6 ' in sub according to the context? yes. i think 'employed by 6 ' in sub is consistent with 'working in 6 ' in src. tion" foot_1 (abbreviated as "h") or a "not hallucination" (abbreviated as "n ") label to the highlighted spans. to simulate real-world nlg applications, we propose two sub-tasks with "offline" and "online" settings. in the offline setting, it is assumed that generation is complete, so the the model is able perceive the bidirectional context. this could be used in the post-generation examination of nlg systems. for online detection, the model can only access the unidirectional preceding context, which simulates on-the-fly generation. online detection is important in practice as it enables nlg systems to proactively forestall potential hallucinations. ## dataset creation to collect the hades dataset, we first perturb "raw text" web data into "perturbed text" (fig 2a ) (sec 3.2). we then ask human annotators to assess whether the perturbed text spans are hallucinations given the original text (fig 2b ) (sec 3.3). ## baseline models as an initial step towards tackling the proposed hallucination detection task and benchmarking methods, we create several baseline detection models foot_9 . feature-based models as elaborated in sec 3.4, the statistical/model-based features like average word probability, average entropy, tf-idf, ppmi, as well as parsing features like pos and ner tags can be vague indicators of hallucinations. the former two are context-aware and the latter four are not. we incorporate them as features to build classifiers including logistic regression (lr) and support vector machine (svm) using scikit-learn @xcite . the maximum number of iteration is set as 100, with an early-stop strategy which stops training if the loss does not drop within 5 iterations. ## experimental setup baseline configurations for the transformerbased baselines, we experiment with a variety of pretrained models via hugging face transformers @xcite , including bertlarge (335m), gpt2-medium (345m), xlnetlarge (340m), roberta-large (355m). we use adam optimizer @xcite with different learning rates, i.e. 5e-3 for gpt2 and bert and 1e-3 for other models. we explored multiple model architectures and setups to determine the optimal configuration using bert-large model. as discussed in sec.2, hades can serve as benchmark for hallucination detection in both offline (model can see bidirectional context) and online (only preceding context can be leveraged) settings. note that we apply the feature-based baselines only in the offline setting ( evaluation metrics we evaluate the baselines on hades with standard classification metrics including accuracy, precision, recall, f1 and auc (area under curve) with respect to roc. we also utilize the g-mean metric which measures geographic mean of sensitivity and specificity @xcite and they were reported useful especially for the imbalanced label distribution scenarios. we also employ the brier skill score (bss) metric @xcite , which calculates the mean squared error between the reference distribution and the hypothesis probabilities. ## results baseline performance figure 5 : the visualization of predicted hallucination scores for a sample of gpt-3 generated text, provided by bert (large, offline) detector. darker green signifies higher risk to be hallucinations. presumably, the causal language model pretraining method makes gpt-2 perform better in the autoaggressive (online) detection setting. context matters in hades to investigate extent to which contextual information helps the hallucination detection in hades, we run bert-large detection model with different context lengths and characterize its performance in both online and offline settings in fig 6 . starting from the target words, we set a fixed size (5/10/20/40/80/160) context window and truncate all text beyond this window. as we enlarge the context window, model performance grows rapidly when context length is smaller than 80, and then gradually converges. this observation highlights the importance of context in hallucination detection. interestingly, we observe that the model obtains higher performance in the offline mode than in the online setting. the performance gap between the two settings maximizes when context length is around 75, and vanishes with long (> 150) or short (< 20) context windows. we surmise that for long (> 150) context window, the preceding context information might already be adequate for detection, while for short (< 20) context windows, the context, regardless whether it is unidirectional or bidirectional, might not contain enough information for detection. ## related work reference-based hallucination detection apart from human verification @xcite , researchers have developed effective reference-based methods which automatically detect hallucination in the generated text using statistical n-gram matching @xcite , edit distance heuristics @xcite , natural language inference @xcite , information extraction @xcite or question answering @xcite @xcite . our approach differs from them in that we investigate the reference-free hallucination detection scenario. to reduce hallucinations in the reference-based setting, researchers have applied iterative training @xcite , post editing @xcite , soft constraints, e.g. attention manipulation @xcite @xcite or optimal transport @xcite , and template/scaffold guided schema with explicit plans @xcite @xcite @xcite , e.g. text sequences which specify the narrative ordering, and implicit plans @xcite @xcite , e.g. (structured) hidden variables that corresponds to certain surface realization. ## conclusions we have proposed a token-level reference-free hallucination detection task and introduced a benchmark dataset hades for identifying fine granularity hallucination in free-form text generation. to create this dataset, we perturbed texts to simulate hallucination in nlg system, and performed an interative model-in-the-loop annotation approach to annotate the perturbed text in an imbalanced label scenario. we have further provided comprehensive analyses of hades and evaluated several baseline models to establish initial benchmarks. we hope that the proposed task and dataset will shed light on high-resolution hallucination detection in freeform text generation and will eventually lead to real-time hallucination prevention.