A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection (2023)

Abstract: Large Language Models (LLMs) have shown their ability to collaborate effectively with humans in real-world scenarios. However, LLMs are apt to generate hallucinations, i.e., makeup incorrect text and unverified information, which can cause significant damage when deployed for mission-critical tasks. In this paper, we propose a self-check approach based on reverse validation to detect factual errors automatically in a zero-resource fashion. To facilitate future studies and assess different methods, we construct a hallucination detection benchmark named PHD, which is generated by ChatGPT and annotated by human annotators. Contrasting previous studies of zero-resource hallucination detection, our method and benchmark concentrate on passage-level detection instead of sentence-level. We empirically evaluate our method and existing zero-resource detection methods on two datasets. The experimental results demonstrate that the proposed method considerably outperforms the baselines while costing fewer tokens and less time. Furthermore, we manually analyze some hallucination cases that LLM failed to capture, revealing the shared limitation of zero-resource methods.

## introduction recent research has shown that large language models (llms), like chatgpt and gpt-4 (ope-nai, 2023), have achieved state-of-the-art across various nlg tasks @xcite @xcite . despite their remarkable capabilities in various aspects, llms still suffer from some inherent drawbacks, underperforming in scenarios involving complex reasoning @xcite and global learning @xcite . one prominent weakness of llms is their tendency to hallucinate when generating fluent and * equal contribution † work done during internship informative responses. this hallucination issue significantly undermines public trust in llms and limits their deployment in certain critical and sensitive domains such as legal @xcite , medical @xcite , and finance @xcite . some recent studies have attempted to alleviate the hallucination of llms. they emphasized that knowledge gaps in llms are a primary cause of hallucination @xcite @xcite , which has inspired a retrieval strategy to mitigate hallucination through interacting with external knowledge bases @xcite . moreover, there is a growing research in zero-resource hallucination detection @xcite . however, in our view, previous studies suffer from the following two disadvantages: (1) relying on external retrieval modules. retrieving external knowledge often has complex processes and notable delays. in addition, external databases may also be inaccessible. (2) only focusing on sentence-level hallucination detection. real-world applications often require passage-level hallucination detection rather than sentence-level detection. this arises from the fact that llms tend to furnish users with comprehensive and informative answers instead of a single sentence. in many scenarios, a judgment about the entire passage is enough, which enables a quick decision on whether to activate the retrieval module and generate a new response. it's highly inefficient and costly to access the truthfulness of a response sentence by sentence. therefore, exploring passage-level hallucination detection holds greater practical significance than exclusively concentrating on sentence-level detection. unfortunately, research on passage-level hallucination detection is still scarce: to the best of our knowledge, there is no available method and suitable benchmark currently. to facilitate research on passage-level hallucination detection, we first annotate and propose the phd (passage-level hallucination detection) benchmark, a new benchmark for the evaluation of passage-level detection methods. we then replicate some recent sentence-level detection methods and adapt them to passage-level detection. due to the poor performance of these methods on our benchmark, we propose a zero-resource hallucination detection method named reverse validation (rv). rv is specifically tailored for passage-level detection, which can detect whether a passage contains factual errors without access to token-level probability distribution and external knowledge. we empirically evaluate the rv method on the phd benchmark and the wikibio-gpt3 dataset @xcite . the results show that the rv method significantly outperforms existing methods and baselines, demonstrating competitive performance, lower token costs, and faster response times. in addition, we conduct an ablation study to explore the robustness and compatibility of our approach for different llms. finally, we reveal the shared limitation of zero-resource detection methods by investigating bad cases. we are committed to facilitating research on passage-level hallucination detection. our contributions can be summarized as follows: (1) we create a high-quality benchmark named phd for the evaluation of passage-level hallucination detection methods, and it will become a challenging benchmark for hallucination detection. (2) we propose the reverse validation method to detect passage-level hallucinations, which can be used in black-box models and zero-resource fashion. (3) we demonstrate the effectiveness and robustness of the proposed method by comparing it with existing methods on two datasets. data and code will be released at https://github.com/maybenotime/phd . ## the phd benchmark in this section, we describe the detailed information of our phd benchmark. specifically, we show how we select entities to generate hallucination data in section 3.1. then, we give the annotation process of phd benchmark in section 3.2. finally, we report statistical information about our phd benchmark in section 3.3. ## experimental setup in this section, we introduce the model, the datasets, the evaluation metrics, and the methods that are compared in the experiments. ## case study and analysis we manually analyzed the false negative and false positive examples of our method to give a better accuracy min hal. ratio lmvslm revised 29.8% 60.0% zero-shot detect. 0% n/a rv via qg 74.2% 10.0% rv via em 67.2% 10.0%