H allu M easure: Fine-grained Hallucination Measurement Using Chain-of-Thought Reasoning (2024)

Abstract: Automating the measurement of hallucinations in LLM generated responses is a challenging task as it requires careful investigation of each factual claim in a response. In this paper, we introduce HalluMeasure, a new LLM-based hallucination detection mechanism that decomposes an LLM response into atomic claims, and evaluates each atomic claim against the provided reference context. The model uses a step-by-step reasoning process called Chain-of-Thought and can identify 3 major categories of hallucinations (e.g., contradiction) as well as 10 more specific subtypes (e.g., overgeneralization) which help to identify reasons behind the hallucination errors. Specifically, we explore four different configurations for HalluMeasure’s classifier: with and without CoT prompting, and using a single classifier call to classify all claims versus separate calls for each claim. The best-performing configuration (with CoT and separate calls for each claim) demonstrates significant improvements in detecting hallucinations, achieving a 10-point increase in F1 score on our TechNewsSumm dataset, and a 3-point increase in AUC ROC on the SummEval dataset, compared to three baseline models (RefChecker, AlignScore, and Vectara HHEM). We further show reasonable accuracy on detecting 10 novel error subtypes of hallucinations (where even humans struggle in classification) derived from linguistic analysis of the errors made by the LLMs.

## introduction hallucinations in large language models (llms) are inevitable @xcite and can cause significant harm. for instance, non-existent legal cases generated by an llm in court papers submitted by a law firm resulted in sanctions by a judge @xcite . wrongly claiming the webb space telescope was the first to photograph an exoplanet during an llm-based product demo led to a 7.7% drop * these authors contributed equally to this work. in a company's stock value, wiping out $100 billion in market capitalization @xcite . moreover, misleading information about bereavement travel policy generated by an airline chatbot led to a court order for partial refund to a passenger @xcite . consequently, detecting and measuring hallucinations in llms has become a crucial research area in recent years to prevent their potential harmful effects. we measure hallucinations in llm responses by comparing the claims made in a response against a reference context document. since manually annotating llm responses against such context documents is time-consuming and expensive, there is a need to develop automatic approaches to measure hallucination at scale. in recent years, several studies @xcite @xcite @xcite have proposed solutions to automate measurement of hallucinations in llm responses. these solutions involve classifying pairs of context and llm response texts into hallucination vs. non-hallucination classes using machine learning models. we highlight key differences between prior solutions and our hallumeasure method below: llm-in-test response atomic claims samsung's gear blink could have a projected keyboard that allows you to type in the air. ralph lauren's polo tech shirt uses biosensing fabrics to monitor physical activity. hush earplugs filter out unwelcome sounds while allowing phone calls and alarms to intrude. 1. samsung has a product called gear blink. 2. gear blink could have a projected keyboard. 3. gear blink's projected keyboard would allow typing in air. 4. ralph lauren has a product called polo tech shirt. 5. polo tech shirt uses bio-sensing fabrics. 6. polo tech shirt bio-sensing fabrics monitor physical activity. 7. there is a product called hush earplugs. 8. hush earplugs filter out unwelcome sounds. 9. hush earplugs allow phone calls to be heard. 10. hush earplugs allow alarms to be heard. our hallumeasure method works by first decomposing the llm response into a set of claims using a claim extraction model. then, we classify the claims into 5 key classes (e.g., supported, absent, contradiction, partially supported, and unevaluatable) by comparing them against the contexts using our claim classification model. additionally, we classify the claims into 10 novel distinct error types (e.g., entity, temporal, over-generalization, etc.) that provide a fine-grained analysis of hallucination errors. finally, we produce an aggregated hallucination score by measuring the rate of unsupported claims (i.e., those assigned classes other than supported), and calculate the distribution of fine-grained error types. this distribution provides llm builders with valuable insights into the nature of errors their llm is making, facilitating targeted improvements. figure 1 illustrates the main components and process behind hallumeasure. our results demonstrate that hallumeasure outperforms existing solutions in terms of f1 score and auc roc metric on two datasets: technews-summ (our own curated dataset) and a popular public benchmark dataset summeval @xcite . we attribute hallumeasure's superior performance to (1) our improved prompting strategy that utilizes chain-of-thought (cot) reasoning, and (2) our claim-level classification approach that measures hallucination based on fine-grained information present in the response text. we attempt to answer the following 6 key research questions as part of this study: our key contributions are (1) a novel hallumeasure method that automatically measures hallucinations using fine-grained analysis of llm responses using chain-of-thought reasoning, (2) experimental results of our hallumeasure method that outperforms existing solutions (refchecker, alignscore, and vectara hhem), and (3) a novel technews-summ dataset containing fine-grained claim level labels for news summarization task with tech news articles taken from cnn/dailymail dataset. ## related work hallucination is a topic of growing research interest, and a range of prior studies have addressed its identification and measurement. several survey papers provide a useful overview and analysis @xcite @xcite . a number of works provide either a hallucination measurement dataset @xcite @xcite , an automatic evaluation metric @xcite @xcite @xcite @xcite or a meta-evaluation (i.e., evaluation of different hallucination metric performances) @xcite . many of the use cases addressed focus on summarization (e.g., news summarization or headline generation) and use popular news datasets for testing (cnn/dailymail news articles corpus @xcite , or xsum news headline dataset @xcite ). another popular use case is wikipedia-style biography generation (e.g., as in wikibio dataset @xcite ). previous approaches to measurement include using pretrained or finetuned models or nli-and question-answer-generation (qag)-based metrics. more recent studies employ llms to classify responses. most of these @xcite classify at the response level while a smaller number classify at the sentence @xcite or fine-grained claim level @xcite . further, they differ in whether they use a binary hallucination/non-hallucination classification, ternary nli classes @xcite , or perform fine-grained multi-class classification to divide hallucination into different error types (e.g., negation error, number swap, or entity swap, etc.) @xcite . a number of different taxonomies of hallucination error types have been proposed @xcite @xcite . we build on these earlier approaches by combining claim-level analysis with fine-grained error types (distinguished based on their impact and potential causes or mitigations), and by using current llms with few-shot learning and cot prompting to produce strong results in measuring hallucinations at both the claim and response levels. ## method in this section we present methodology for hal-lumeasure. we use a claim-level hallucination measurement approach inspired by @xcite , @xcite . we decompose llm responses into smaller units ('claims') for more precise measurement using our claim extraction model based on claude 2.1. then, we classify claims and assign high-level labels, further identifying 10 more specific types of hallucination errors for unsupported claims using cot reasoning with claude 3 sonnet (see hallumeasure calculates the percentage of hallucinations at both the claim and response level, and provides the distribution of hallucination types identified. below, we discuss specific components of our hallucination measurement method that takes in context and response pairs as input and produces hallucination scores as output. figure 1 shows the hallumeasure process. ## conclusion we introduce hallumeasure, a novel approach to automatically measure hallucinations in llm responses. hallumeasure decomposes an llm response into set of claims using a claim extraction model based on claude with few-shot prompting. it compares the extracted claims against a context document using a claim classification model leveraging few-shot chain-of-thought prompting with claude to enhance classification performance. an aggregated response-level score is produced by measuring the rate of unsupported claims and distribution of specific error types. we demonstrate the effectiveness of hallumeasure on: tech-newssumm (our own curated dataset with detailed claim-level error labels) and summeval (a popular benchmark dataset). our results demonstrate hal-lumeasure's superior performance over baseline models, with at least a 10-point f1 score improvement on technewssumm and a 3-point auc roc increase on summeval. for future work, we plan to employ dynamic few-shot prompting and use optimized prompts with fast-inference llms.