Knowledge-Centric Hallucination Detection (2024)

Abstract: Large Language Models (LLMs) have shown impressive capabilities but also a concerning tendency to hallucinate. This paper presents RefChecker, a framework that introduces claim-triplets to represent claims in LLM responses, aiming to detect fine-grained hallucinations. In RefChecker, an extractor generates claim-triplets from a response, which are then evaluated by a checker against a reference. We delineate three task settings: Zero, Noisy and Accurate Context, to reflect various real-world use cases. We curated a benchmark spanning various NLP tasks and annotated 11k claim-triplets from 2.1k responses by seven LLMs. RefChecker supports both proprietary and open-source models as the extractor and checker. Experiments demonstrate that claim-triplets enable superior hallucination detection, compared to other granularities such as response, sentence and sub-sentence level claims. RefChecker outperforms prior methods by 18.2 to 27.2 points on our benchmark and the checking results of RefChecker are strongly aligned with human judgments.

## introduction large language models (llms) have sparked a revolution in natural language processing (nlp), covering diverse tasks with a unified architecture @xcite . however, llms exhibit a tendency to generate hallucinated contents that can be difficult to discern, posing a potential risk of misleading users. @xcite . consequently, hallucination detection has received increasing attention @xcite @xcite . detecting hallucination is essentially a job of comparing a generated response against a reference. to this end, several challenges remain, in-… the song 'i ran (so far away)' was originally performed by the english new wave band a flock of seagulls in 1983. the song is 'i ran (so far away)'. 'i ran (so far away)' was originally performed. 'i ran (so far away)' was originally performed by a flock of seagulls. a flock of seagulls is an english new wave band. a flock of seagulls performed 'i ran (so far away)' in 1983. ## related work we undertake a review of prior work relevant to our study and compare them with refchecker. the comparative analysis with three representative methods is encapsulated in hallucination in llms. hallucinations, which frequently occur in nlp tasks like summarization @xcite , machine translation @xcite , dialog systems @xcite and rag @xcite , can be categorized to factuality hallucinations and faithfulness hallucinations @xcite . factuality hallucinations involve claims contradicted by real-world facts, while faithfulness hallucinations are inconsistent with the input content. recent research on hallucination detection primarily concentrates on factuality hallucinations, such as selfcheck-gpt @xcite , factscore @xcite and factool @xcite . we address both factuality and faithfulness hallucinations and further categorizing them into three contextual settings to align with real-world use cases. ## preliminaries hallucinations are claims made by llms not supported by factual knowledge, which we refer to as references; detecting hallucinations involves comparing the claims against the references. this process depends on context settings, the granularity of checking and the definition of hallucinations. we discuss them in turn. context settings. we differentiate three context settings covering various tasks and employ different benchmarks for each setting as shown in figure 3 . zero context (zc) tasks in this setting can be referred to as closed-book question answering which requires the llm to respond solely based on its internal knowledge. therefore, in principle, references should be in the training corpus. in practice, for benchmarking purposes, we use a "ground truth" reference for each question which contains the answer, and we expect the reference can be retrieved from a trusted knowledge source when deployed to real-world applications. noisy context (nc) in this setup, the llm receives additional context retrieved from some external knowledge source, which may contain noisy or irrelevant information. nc is also known as rag, a crucial use case frequently encountered in real-world applications. accurate context (ac) this setting is similar to nc but the reference is typically noise-free. examples include text summarization, closed-qa and information extraction tasks. the main difference between ac and nc is that the context in ac is trustworthy, while the context in nc contains a lot of noise. ## the refchecker framework as illustrated in figure 2 , the refchecker framework is designed as a 2-stage pipeline: an extractor e decomposes the llm response into a set of triplets, with each of them verified by the checker c. the categorization of the triplets can be optionally aggregated according to specified rules. we explain them in the subsequent subsections. extractor our checking framework hinges on a key assumption: the decomposition of the original text into triplets facilitates finer-grained detection and more accurate evaluation. the extraction of these triplets plays a pivotal role in achieving this objective. we apply llms to extract knowledge triplets from the given text. we began with gpt-4 and, for both cost and efficiency concern, mixtral checker we experimented with two families of checkers, the first is off-the-shelf llms, gpt-4 (see appendix b.2 for prompts), and the second is smaller nli models including alignscore @xcite and roberta-nli. 2 long references in ac/nc setting are split to fit into small context windows of these small models (e.g. 200 tokens), and the results are aggregated later. mistral 7b @xcite lies in between, offering both massive knowledge obtained during pre-training and the opportunity for tuning the open model weights with nli data. there are many options we have experimented: 1) fine-tune by adding small amount of new parameters using lora (lora-sft) @xcite , 2) attach a shallow classifier, eg. svm, 2-layer mlp, knn after nca projection @xcite , on top of the internal states of the model. we call such checker repc (for representation-based classifier). such states can be selected from one layer (layer selection, ls) or an ensemble of all layers (layer ensemble, le). as we will report in sec. 6.4, repc checkers are competitive in general. aggregation triplet results can be aggregated to obtain the ratio of each category, therefore gives an overall measure of hallucination distribution in a response. to derive the performance of a particular llm, we take a macro average on entailment/neutral/contradiction ratios of all responses. if a scalar is preferred, we can assign certain numeric values to the catogories, for instance -1, 0, 1 for contradictory, neutral and entail, respectively. the aggregation can be customized and this is one of the benefits of the fine-grained hallucination design in refchecker. for instance, to compare against other response-level approaches (cf. sec. 6.1), we adopt a rule where the response is flagged as contradictory if any one of the claim triplet is contradictory. ## the knowhalbench dataset we assembled a benchmark dataset comprising 300 examples from public datasets, with 100 for each of the three context settings mentioned in section 3. we further collected responses from 7 llms on the established benchmark for human evaluation. ## experiments the major difference between refchecker and other related work lies in the claim granularity. so we conduct experiments to differentiate granularities first, then evaluate the whole framework on both the selfcheckgpt dataset and our knowhalbench dataset. ## conclusion we introduced refchecker, a unified framework for detecting hallucination in llm responses. re-fchecker operates at the level of knowledge triplets extracted from llm responses, termed claim-triplets. it allows for fine-grained hallucination detection. experiments on both previous benchmark and our curated knowhalbench dataset show that such knowledge-centric approach yields superior performance compared to prior work based on surface text (response, sentence, sub-sentence, etc.).