ANAH : Analytical Annotation of Hallucinations in Large Language Models (2024)

Abstract: Reducing the ‘ \textit{hallucination} ' problem of Large Language Models (LLMs) is crucial for their wide applications. A comprehensive and fine-grained measurement of the hallucination is the first key step for the governance of this issue but is under-explored in the community.Thus, we present \textbf{ANAH} , a bilingual dataset that offers \textbf{AN} alytical \textbf{A} nnotation of \textbf{H} allucinations in LLMs within Generative Question Answering.Each answer sentence in our dataset undergoes rigorous annotation, involving the retrieval of a reference fragment, the judgment of the hallucination type, and the correction of hallucinated content. ANAH consists of ~12k sentence-level annotations for ~4.3k LLM responses covering over 700 topics, constructed by a human-in-the-loop pipeline.Thanks to the fine granularity of the hallucination annotations, we can quantitatively confirm that the hallucinations of LLMs progressively accumulate in the answer and use ANAH to train and evaluate hallucination annotators. We conduct extensive experiments on studying generative and discriminative annotators and show that, although current open-source LLMs have difficulties in fine-grained hallucination annotation, the generative annotator trained with ANAH can surpass all open-source LLMs and GPT-3.5, obtain performance competitive with GPT-4, and exhibits better generalization ability on unseen questions.

## introduction large language models (llms) have achieved significant performance improvements across a diverse array of natural language processing tasks @xcite @xcite @xcite . however, llms still face a worrisome problem that significantly hinders their real-world applications, hallucination, in which they produce plausible-sounding but unfaithful or nonsensical information @xcite when answering the user questions, especially those require intensive knowledge. given the fluency and convincing nature of the responses produced by llms, the detection of their hallucinations becomes increasingly difficult @xcite @xcite . such a challenge impedes the deep analysis and reduction of llm hallucination and leads to extensive dissemination of misleading information as the user base widens and real-world applications proliferate @xcite . there have been extensive efforts on effectively detecting and evaluating hallucination @xcite @xcite . however, most benchmarks were proposed before the advent of llm and targeted specific english tasks @xcite , which are not challenging for current models. recent benchmarks @xcite for llms only categorize whether the entire response contains hallucinations without explanation and reference. this coarse-grained nature makes it difficult to trace the exact trigger of hallucinations and obstructs further mitigation of them. therefore, we establish a novel large-scale chinese-english benchmark, named anah 2 , that assesses the llms' ability to annotate the llm hallucinations sentence-by-sentence, in the scenario of knowledge-based generative question answering. rather than solely result-oriented, for each answer to a question, our approach prompts the model to annotate hallucination for each sentence, including retrieving reference fragment for the sentence, judging the hallucination type (no/contradictory/unverifiable hallucinations, and no fact), and correcting the sentence based on the reference fragment if hallucination exists (fig. 1 ). to facilitate the scale-up of datasets, we ensure the comprehensiveness and diversity of anah across various topics, questions, and answers. as shown in fig. 2 , first, we curate topics in both english and chinese, encompassing a broad domain range including things, places, people, and historical events (fig. 3 ). second, we craft around three related questions for each topic to ensure originality and avoid contamination. third, for each question, we construct a high-quality and a low-quality response with and without reference in generation, respectively, enabling a comparative analysis of hallucination distributions across different response scenarios. the final and pivotal stage is fine-grained hallucination annotation, as exemplified in fig. 1 . eventually, we form ∼12k hallucination annotations of ∼4.3k answers to ∼2.2k questions spanning a broad domain range, which is challenging for hallucination detection. thanks to the completeness and fine-granularity of anah, the statistical results of the hallucination annotations quantitatively confirm that hallucinations progressively accumulate in the llm responses. furthermore, anah can be used to 2 anah is short for analytical annotation of hallucinations. train and evaluate hallucination annotators. we first discovered that only gpt-4 could do this task well. thus, we further investigate training generative and discriminative hallucination annotators using anah and observe the advantages of generative annotators over discriminative annotators in handling the imbalance issue of hallucination types. remarkably, our generative annotators achieve an accuracy of 81.01%, surpassing open-source models and rivaling gpt-4 (86.97%) in performance with a smaller size and lower source cost. we also observe that the hallucination annotators consistently exhibit better generalization regarding the number of questions than the breadth of topics, thereby guiding us toward prioritizing data scaling to cover a broader array of topics in future research. ## dataset construction anah's establishment contains four stages (fig. 2 ): (1) selecting a broad range of topics to ensure comprehensiveness ( § 2.1), ( 2 ) constructing related questions whose responses can be fully supported by reference ( § 2.2), (3) generating answers from llms under different models and scenarios ( § 2.3), and (4) fine-grained hallucination annotation for further analysis and mitigation ( § 2.4). ## hallucination annotator taking advantage of the rich fine-grained annotations in anah, we explore training and evaluating both generative and discriminative annotators. the generative annotator generates textual annotations including reference fragments, hallucination type, and correction; while the discriminative annotator only focuses on discriminating hallucination type. ## conclusion and future work hallucinations in generative tasks present substantial obstacles to the reliability and creditability of llms but lack a comprehensive and fine-grained detecting strategy. thus, we present a bilingual dataset, anah for fine-grained hallucination annotation in gqa covering diverse topics, offering the opportunity to quantitatively analyze hallucination phenomena such as accumulation effect, and facilitating the development of state-of-the-art finegrained hallucination annotators. our generative hallucination annotators surpass all open-source llms and gpt-3.5 and obtain performance on par with gpt-4. our generalization experiments indicate that improving the breadth of topics in the dataset is more important than extending questions under existing topics in the dataset. this paper paves the way for further scaling up the dataset of anah to conduct a systematic evaluation and analysis of llm hallucinations, with the trained hallucination annotators. the hallucination annotators also have the potential to be used in the hallucination mitigation pipeline in both the training and inference stages.