Unified Hallucination Detection for Multimodal Large Language Models (2024)

Abstract: Despite significant strides in multimodal tasks, Multimodal Large Language Models (MLLMs) are plagued by the critical issue of hallucination. The reliable detection of such hallucinations in MLLMs has, therefore, become a vital aspect of model evaluation and the safeguarding of practical application deployment. Prior research in this domain has been constrained by a narrow focus on singular tasks, an inadequate range of hallucination categories addressed, and a lack of detailed granularity. In response to these challenges, our work expands the investigative horizons of hallucination detection. We present a novel meta-evaluation benchmark, MHaluBench, meticulously crafted to facilitate the evaluation of advancements in hallucination detection methods. Additionally, we unveil a novel unified multimodal hallucination detection framework, UNIHD, which leverages a suite of auxiliary tools to validate the occurrence of hallucinations robustly. We demonstrate the effectiveness of UNIHD through meticulous evaluation and comprehensive analysis. We also provide strategic insights on the application of specific tools for addressing various categories of hallucinations.

## introduction the recent emergence of mllms @xcite @xcite that more closely mirror human cognition and learning has unleashed unprecedented possibilities for the future of artificial general intelligence (agi). despite mllms' impressive abilities, they are susceptible to generating seemingly credible content that contradicts input data or established world knowledge, a phenomenon termed "hallucination" @xcite @xcite which team does the athlete on the right side in the below picture belong to ? s1 [the athlete on the right side, wearing the red uniform in the image, belongs to the american soccer team club américa.], s2 [the scene is filled with the excitement of a soccer match.] s1.1: the athlete on the right side wears the red uniform. s1.2: the athlete on the right side belongs to club américa. s1.3: club américa is the american soccer team. s2.1. the scene is filled with the excitement of a soccer match. (a) image-to-text text detect claims from response detect claims from user query to identify and detect modality-conflicting hallucinations at various levels such as object, attribute, and scene-text, as well as fact-conflicting hallucinations in both image-to-text and text-to-image generation. our benchmark emphasizes finegrained detection, with "s1" representing the segment and "s1.1" and "s1.2" denoting its corresponding claims. @xcite . these hallucinations hinder the practical deployment of mllms and contribute to the dissemination of misinformation. consequently, detectors that could detect multimodal hallucinations @xcite within responses from mllms are urgently needed to alert users to potential risks and drive the development of more reliable mllms. although several works have been conducted to detect hallucinations from mllms @xcite @xcite or alleviate hallucinations @xcite , these efforts operate in isolation and have certain limitations when compared with the aspects illustrated in figure 1: (1) task singularity: current research has primarily concentrated on specific tasks, such as image captioning while neglecting that text-to-image generation, an important component of agi, also suffers from hal-lucinations induced by mllms. (2) limited hallucination categories: prior studies have focused on identifying hallucinations at the object level, yet they fail to consider the prevalence of scene-text or factual inconsistencies that also frequently occur in mllms. (3) incomplete granularity: it would be more valuable to assess hallucinations at a finegrained level, examining individual claims within a response, rather than evaluating the entire response holistically. considering these constraints hinder rapid progress in practical hallucination detection, it raises the question: can we develop a unified perspective for detecting hallucinations from mllms? to further investigate this problem, we have broadened the concept of multimodal hallucination within mllms to a holistic framework, integrating both image-to-text generation such as image captioning (ic) and visual question answering (vqa), as well as text-to-image-synthesis (t2i) -to align with mllms' capabilities of performing varied multimodal tasks. we are committed to exploring a broad spectrum of hallucinatory categories and the intricate nuances of claim-level hallucination through a lens that integrates both modality-conflicting and fact-conflicting hallucinations. based on the outlined perspectives, we have developed the multimodal hallucination detection benchmark (mhalubench) to assess the progress of unified multimodal hallucination detectors for mllms and embodied the data framework depicted in figure 1 . at its core, leveraging mllms' inherent selfdetection mechanisms to pinpoint diverse hallucinations encounters significant hurdles. we further develop a tool-augmented framework for unified hallucination detection, named unihd, which integrates evidence from multiple auxiliary tools through the following procedure: (1) essential claim extraction involves extracting the core claims within the generated response for image-totext generation or user queries in text-to-image generation; (2) autonomous tool selection via query formulation prompts mllms (gpt-4/gemini) to autonomously generate pertinent questions for each claim. these questions are crafted to determine the specific type of tool required for each claim and to establish the input for the tool's operation; (3) parallel tool execution deploys a suite of specialized tools to operate concurrently, providing evidence from their outputs to reliably validate potential hallucinations; (4) hallucination verification with rationales aggregates the collected evidence to in-struct the underlying mllm to judge whether the claim hallucinatory with rationals for explanation. we have conducted a thorough evaluation of the unihd framework, utilizing the underlying mllm against the mhalubench benchmark. our findings underscore the effectiveness of our approach and confirm that multimodal hallucination detection remains a formidable challenge. in a nutshell, we conclude our contributions as: text-to-image ... 1. whether the output text contradicts the information presented in the input image. 2. whether the output text conflicts with world knowledge. 1. whether the output image contradicts the information presented in the input text. 2. whether the output image conflicts with the world knowledge underlying the text. ## preliminaries we explore a unified perspective on hallucination in mllms (illustrated in figure 2 ) with the aspiration of developing a unified detection framework. unified view of multimodal hallucination taxonomy. a prerequisite for unified detection is the coherent categorization of the principal categories of hallucinations within mllms. our paper superficially examines the following hallucination taxonomy from a unified perspective: datasets response purpose granularity hallucination types modality scenario generated by object attribute scene text fact task factcc (kryscinski et al., 2020) synthetic check. sentence ✔ text text2text qags (wang et al., 2020) model check. summary ✔ text text2text halueval (li et al., 2023a) chatgpt det. response ✔ text text2text pope (li et al., 2023b) -eval. response ✔ multi. image2text haelm (wang et al., 2023c) -det. response multi. image2text amber (wang et al., 2023b) -eval. response ✔ ✔ multi. image2text mhalubench (ours) mmlms det. res.,seg.,claim ✔ ✔ ✔ ✔ multi. image2text/text2image ## construction of mhalubench to facilitate research in this area, we introduce the meta-evaluation benchmark mhalubench, which encompasses the content from image-to-text and text-to-image generation, aiming to rigorously assess the advancements in multimodal hallucination detectors. our benchmark has been metic-ulously curated to include a balanced distribution of instances across three pivotal tasks, which encompasses 200 exemplars for the task of ic 200 for vqa, and an additional 220 dedicated to text-to-image generation. the comparison of mhalubench with other benchmarks is detailed in ## unihd: unified hallucination detection framework for mllms we present unihd in figure 5 and follow. the specific prompts are listed in appendix a ## conclusion we introduce a unified problem formulation for multimodal hallucination detection that encompasses a diverse range of multimodal tasks and hallucination types. a fine-grained benchmark dataset, mhalubench, is also proposed to promote this challenging direction. alongside this, we present the unified hallucination detection framework, unihd, capable of autonomously selecting external tools with capturing pertinent knowledge to support hallucination verification with rationales. our experimental results indicate that unihd achieves better performance across both image-to-text and text-to-image generation tasks, confirming its universality and efficacy.