Enhancing Hallucination Detection through Perturbation-Based Synthetic Data Generation in System Responses (2024)

Abstract: Detecting hallucinations in large language model (LLM) outputs is pivotal, yet traditional fine-tuning for this classification task is impeded by the expensive and quickly outdated annotation process, especially across numerous vertical domains and in the face of rapid LLM advancements. In this study, we introduce an approach that automatically generates both faithful and hallucinated outputs by rewriting system responses. Experimental findings demonstrate that a T5-base model, fine-tuned on our generated dataset, surpasses state-of-the-art zero-shot detectors and existing synthetic generation methods in both accuracy and latency, indicating efficacy of our approach.

## introduction large language models (llms) tend to produce hallucinations, wherein the generated text either contradicts the given source knowledge (intrinsic hallucination) or cannot be verified against it (extrinsic hallucination) @xcite . despite the burgeoning enthusiasm for deploying generative ai and llms in real-world applications, the issue of hallucinations poses significant concerns for downstream users. consequently, the detection of hallucinations is paramount in enhancing the safety of llm applications and in fostering trust among users of these technologies. an effective hallucination detection system should be accurate, fast, and affordable. costeffectiveness is crucial because every check for hallucinations adds extra cost to the use of large language models (llms), which may already be substantially high. moreover, the system must possess the flexibility to adapt to the rapidly evolving landscape of llms. as shown in unfortunately, many current methodologies are either i) costly in terms of compute @xcite or ii) depend on out-ofdomain/external resources such as qa @xcite or nli annotation @xcite , potentially compromising performance. in this study, we introduce a simple yet effective approach for automatically generating synthetic annotations to train hallucination detectors. figure 1 shows an overview of our approach. the core of our method involves prompting a rewriting llm to transform a given system response from the target llm into both faithful and hallucinated versions, respectively. this technique distinguishes itself from existing methods @xcite @xcite in three significant ways. first, unlike traditional methods that rely on human-annotated examples of faithfulness, our strategy is entirely automated, eliminating need for manual annotation. second, by directly altering responses from the target llm, our trained detector aligns more closely with the response distribution of the target llm, facilitating seamless adaptation to new llms. lastly, while previous approaches require predefined information about the types of hallucinations for their generation process, our method operates without such assumptions. this allows for the creation of a broader spectrum of hallucination types, enhancing the coverage and diversity of generated hallucinations. bot: yes, he starred alongside jake gyllenhaal in that one. quite a gripping film! bot: yes, he starred in zodiac, which is a crime fiction film. bot: yes, he starred alongside chris hemsworth in that one. quite a gripping film! system response user: do you like iron man assistant: sure do! robert downey jr. is a favorite user: yes i like him too did you know he also was in zodiac a crime fiction film. iron man is starring robert downey jr.. robert downey jr. starred in zodiac (crime fiction film). zodiac (crime fiction film) is starring jake gyllenhaal faithful response hallucination response dialogue history knowledge iii. finetuning ii. rewriting i. sampling our experimental evaluations span two hallucination detection datasets, opendialkg @xcite and begin @xcite , where a t5base model, fine-tuned with our novel data generation approach, significantly surpasses gpt-4 based methods in performance while achieving a tenfold increase in speed. further analysis of the generated hallucinations uncovers previously unreported patterns, such as "adding attributes", expanding the discourse on hallucination beyond existing literature. our code and data will be available at https: //github.com/asappresearch/halugen . ## methodology in this section, we detail our methodology for generating synthetic hallucinations that closely mimic those encountered in real-world applications of large language models (llms). prior approaches to hallucination generation have primarily relied on rewriting human-authored texts @xcite or introducing perturbations to the knowledge source @xcite @xcite . however, these methods often yield outputs that diverge significantly from those produced by llm systems, leading to a substantial discrepancy between the synthetic hallucinations and the genuine hallucinations observed in practice. to address this gap, our approach involves prompting a rewriting llm to perturb the responses of the llm system itself, rather than those written by humans. this strategy draws inspiration from the "minor perturbation" technique described by @xcite , adapted to our context to ensure the synthetic hallucinations closely align with the expected data in real-world deployments. to effectively train a hallucination detector, it is imperative to have access to both hallucinated and faithful responses. unlike previous studies, where human-curated outputs served as the benchmark for faithful system outputs @xcite @xcite , the responses obtained directly from the target llm system may contain a considerable proportion of non-faithful responses. to overcome this challenge, we employ the rewriting llm to adjust the system's responses in a manner that promotes the generation of faithful outputs. the specific prompts utilized for inducing both hallucination and faithfulness are presented in appendix §a. it is important to note that our process for generating hallucinations did not involve biasing the system with predefined categories of hallucination within the prompt, ensuring a more authentic and unbiased generation process. foot_0 for the rewriting llm, we selected gpt-4 foot_1 due to its robust capabilities in text rewriting @xcite . leveraging a powerful rewriting llm like gpt-4 enables the exploration of a wider array of hallucination categories, thereby enhancing the coverage of hallucinations that are likely to be encountered in real-world scenarios. begin is a knowledge-grounded dialog dataset featuring 12k responses from four dialogue systems distributed over 3 document-scale knowledge domains -wizard of wikipedia @xcite , topicalchat @xcite and dog @xcite ) -all with mean knowledge snippets longer than opendialkg. in addition, there are three response categories in begin: fully attributable, not fully attributable, generic. generic category refers to response that are vague and do not provide any new information. therefore, in addition to faithful and hallucination generation, we also ask llm to generate responses under "generic" category. the detailed prompt can be found in appendix a ## related work research on generating synthetic annotations for hallucination detection has explored various strategies. some approaches, like fade @xcite and halueval @xcite , manipulate human-written texts by altering entities or applying predefined hallucination criteria, respectively. these methods only focus on introducing hallucinations and ignore faithfulness augmentation, assuming human-generated content to be inherently accurate, which maybe untrue. other studies focus on modifying the knowledge source before response generation. augwow @xcite introduces hallucinations by using irrelevant or no evidence, while begin-adv @xcite alters subjects, objects, named entities, or verbs in the source material, prompting a gpt2-based system @xcite for response regeneration. these techniques, however, might lead to predictable hallucination patterns due to their reliance on predefined rules. more recently, icd @xcite mitigates llm hallucinations by finetuning a model on non-factual samples, aiming to down-weight factually weak predictions. despite its novelty, the reliance on entity perturbation for generating nonfactual samples could limit the coverage of detected hallucinations. halueval-wild @xcite aims to evaluate llm hallucination in human-llm interactions. their approach first collects challeng-ing user queries which can lead to hallucinated llm responses. faithful reference responses are generated using gpt4 with retrieval augmentation. while the generated data is challenging, it is not obvious how to adapt the approach for customized tasks. interestingly, @xcite observed that the effectiveness of the llm-generated synthetic data in supporting model training is negatively correlated with the subjectivity of the target task. ## conclusions in this work, we aim to address the prevalent challenge of training data for hallucination detection being either unavailable or expensive to curate. we hypothesize that this can be addressed via a framework that automatically synthesizes both hallucinated and faithful responses using a prompt-based method. our experimental results on two datasets verify effectiveness of our approach and show it compares favourably against several baselines, including those using prompt-based synthesis.