RAG - HAT : A Hallucination-Aware Tuning Pipeline for LLM in Retrieval-Augmented Generation (2024)

Abstract: Retrieval-augmented generation (RAG) has emerged as a significant advancement in the field of large language models (LLMs). By integrating up-to-date information not available during their initial training, RAG greatly enhances the practical utility of LLMs in real-world applications. However, even with RAG, LLMs can still produce inaccurate outputs, such as distorting or misinterpreting source content, posing risks in high-trust scenarios. To address these issues, we introduce a novel approach called Hallucination Aware Tuning (HAT). This method involves training hallucination detection models that generate detection labels and provide detailed descriptions of the detected hallucinations. Utilizing these detection results—particularly the hallucination descriptions—GPT-4 Turbo is employed to correct any detected hallucinations. The corrected outputs, free of hallucinations, along with the original versions, are used to create a preference dataset for Direct Preference Optimization (DPO) training. The fine-tuning through DPO leads to LLMs that exhibit a reduced rate of hallucinations and deliver improved answer quality.

## introduction guided by the principle of scaling up model size and training data @xcite , transformer-based large language models (llms) have achieved significant milestones in various tasks. despite these advancements, llms continue to confront challenges, particularly with issues of hallucination @xcite . the introduction of the retrieval augmented generation (rag) method has not only broadened the applicability of llms @xcite but has also shown effectiveness in mitigating hallucinations @xcite . however, the persistence of hallucination still restricts the advancement of rag systems @xcite . this issue is especially significant in rag-based applications that process real-time data and may have substantial real-world impacts. notably, there have already been attempts to implement rag technology in critical sectors, including finance @xcite and healthcare @xcite . the hallucination problem is attracting increasing attention from both academia and industry, leading to the development of two focused research domains: hallucination detection and hallucination mitigation. researchers have made notable advancements in both fields @xcite @xcite . however, there is a lack of research effectively integrating detection and mitigation models for the reduction of hallucinations. in this paper, we introduce rag-hat, a novel hallucination aware fine-tuning pipeline designed to effectively combine hallucination detection and mitigation. utilizing a rag output as input, this pipeline features a detection model trained to identify hallucinations and provide humanreadable descriptions of these occurrences. the insights from the detection model are subsequently employed to guide @xcite in revising the rag output to remove any hallucinations. following this initial step, both the original and revised rag outputs are paired and used to train the llm being used in the rag setup through direct preference optimization (dpo) @xcite . this method markedly reduces the rates of hallucination and enhances the quality of the responses. in this paper, our key contributions are: ## analysis 6.1 impact of defensive advice and overly cautious penalization (ocp) we conducted a set of ablation experiments on we-bglm to demonstrate the effectiveness of defensive advice as described in section 4.1.1, as well as the data augmentation by random deletion of one sentence from "chosen" examples as described in section 4.3.2. as is illustrated in the model trained on the full dataset achieved a 51.5% win rate, outperforming the model trained without defensive advice, and a 54.1% win rate trained without both. notably, the win rate increased by 2.6% upon the removal of ocp, underscoring the efficacy of our penalization strategy. ## industry application as a local information provider, newsbreak is actively exploring various applications of retrieval-augmented generation (rag) systems within our business model. our primary focus is on using the rag system to gather fragmented local data and leveraging large language models (llms) to organize this information into coherent formats, such as question-answering systems or highly informative resources. we are currently experimenting with the integration of rag-hat into our rag system to enhance the accuracy and relevance of the local information we provide. users can access a wide range of information through our platform, particularly about local entities (e.g., restaurants, auto shops), local safety (e.g., crime reports), and community events. thus, we incorporate substantial amounts of local merchant data, news articles, and other sources into our training datasets. techniques like rag-hat significantly reduce the risk of unintentionally disseminating misinformation, which is critical for protecting our reputation. additionally, they enable our product managers to plan more advanced rag applications with confidence, mitigating potential legal and reputational risks associated with hallucinations. ## conclusion in this work, we introduce a hallucination-aware tuning pipeline rag-hat, which contains three parts: detection, rewriting and mitigation. the detection component identifies hallucinations with human-readable interpretations. the rewriting component allows us to automatically generate a preference dataset, enabling the use of dpo to train models to hallucinate less. specialized data augmentation techniques are designed to reduce hallucinations without compromising the model's answer quality. benchmarks demonstrated that rag-hat significantly reduced the hallucination rate while enhancing answer quality simultaneously.