RAGT ruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2024)

Abstract: Retrieval-augmented generation (RAG) has become a main technique for alleviating hallucinations in large language models (LLMs). Despite the integration of RAG, LLMs may still present unsupported or contradictory claims to the retrieved contents. In order to develop effective hallucination prevention strategies under RAG, it is important to create benchmark datasets that can measure the extent of hallucination. This paper presents RAGTruth, a corpus tailored for analyzing word-level hallucinations in various domains and tasks within the standard RAG frameworks for LLM applications. RAGTruth comprises nearly 18,000 naturally generated responses from diverse LLMs using RAG. These responses have undergone meticulous manual annotations at both the individual case and word levels, incorporating evaluations of hallucination intensity. We not only benchmark hallucination frequencies across different LLMs, but also critically assess the effectiveness of several existing hallucination detection methodologies. We show that using a high-quality dataset such as RAGTruth, it is possible to finetune a relatively small LLM and achieve a competitive hallucination detection performance when compared to the existing prompt-based approaches using state-of-the-art LLMs such as GPT-4. Furthermore, the finetuned model can effectively mitigate hallucination in LLM responses.

## introduction large language models (llms) have achieved remarkable success in a variety of tasks, including text generation @xcite , machine translation @xcite , and question answering @xcite . however, one of the key challenges in deploying llms in real-world applications is their tendency to hallucinate @xcite . hallucination in the context question: how to prepare to get an ultrasound? reference: passage 1: 1 if you are just interested in checking the gender, you can go as early as 17 weeks. 2 any earlier, and the results may not be accurate, leading to an extra coat of paint for your bedroom. 3 to get good pictures, 3d ultrasounds are best performed between 20 and 32 weeks, and the best pictures are between 24 and 30 weeks. ... response: based on the given passages, here's how to prepare for an ultrasound: * for a gender reveal ultrasound, wait until at least 17 weeks of pregnancy to ensure accurate results. * for 3d ultrasounds, schedule the appointment between 20 and 32 weeks of pregnancy for the best pictures ... ## construction process of ragtruth we established a data generation and annotation pipeline as shown in figure 1 . ## conclusion in this paper, we introduce ragtruth, a large-scale corpus of naturally generated hallucinations, featuring detailed word-level annotations tailored for rag scenarios. our work includes an in-depth analysis of the interplay between hallucinations and various factors, such as task types, models being used, and contextual settings. additionally, we conduct empirical benchmarks of several hallucination detection approaches using our corpus. we show that fine-tuning llama with ragtruth leads to competitive performance. this implies that by using a high-quality dataset such as ragtruth, it is possible to develop specialized hallucination detection models that are highly effective when compared to prompt-based methods using general models such as gpt-4. simultaneously, our findings reveal that identifying hallucinations in rag contexts, particularly at the span level, remains a formidable challenge, with current methods still falling short of reliable detection. we hope that ragtruth, can assist the development of hallucination detection techniques for retrieval augmented generation.