Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models (2025)

Abstract: Hallucination issues continue to affect multimodal large language models (MLLMs), with existing research mainly addressing object-level or attribute-level hallucinations, neglecting the more complex relation hallucinations that require advanced reasoning. Current benchmarks for relation hallucinations lack detailed evaluation and effective mitigation, and their datasets often suffer from biases due to systematic annotation processes. To address these challenges, we introduce Reefknot, a comprehensive benchmark targeting relation hallucinations, comprising over 20,000 real-world samples. We provide a systematic definition of relation hallucinations, integrating perceptive and cognitive perspectives, and construct a relation-based corpus using the Visual Genome scene graph dataset. Our comparative evaluation reveals significant limitations in current MLLMs’ ability to handle relation hallucinations. Additionally, we propose a novel confidence-based mitigation strategy, which reduces the hallucination rate by an average of 9.75% across three datasets, including Reefknot. Our work offers valuable insights for achieving trustworthy multimodal intelligence. The dataset and code are released at https://github.com/JackChen-seu/Reefknot.

## introduction in recent years, large language models (llms) have revolutionized the ai field by expanding their training data to trillions of tokens and increasing their parameter counts to hundreds of billions @xcite @xcite . this has unlocked powerful emergent abilities, and seen widespread applications in diverse domains @xcite @xcite . recently, the community managed to combine visual backbones with powerful llms, resulting in multimodal large language models (mllms) @xcite . while this has led to advancements in multimodal scenarios, it also presents challenges for mllms, notably their tendency to generate hallucinations @xcite . in llms, hallucinations occur when the model generates inaccurate or misleading factual information that cannot be supported by existing knowledge @xcite . however, such issues become more complex in mllms, as hallucinations can manifest as responses containing references or descriptions of the input image that are incorrect @xcite . therefore, it is crucial to evaluate and mitigate these hallucinations to improve the trustworthy of mllms in real-world scenarios. hallucination in mllms likely originates from knowledge biases between pre-training and finetuning, including statistical biases in the training data, over-reliance on parametric knowledge, and skewed representation learning, as suggested by previous research @xcite . specifically, the hallucination in mllms can be divided into three categories: object, attribute and relation hallucinations @xcite . despite these efforts, the community barely considers relation hallucination, which demands more complex reasoning capabilities from mllms. figure 1 (c) reveals that relation hallucination is related to at least two objects simultaneously in the image, through either perceptive or cognitive perspective. furthermore, as outlined in specifically, these works either adopted either simple yes/no (y/n) task to assess the models' accuracy/precision, or utilized llm to score the performance, but none of them were able to give a comprehensive evaluation from both discriminative (y/n mcq) and generative (vqa) perspectives. moreover, previous benchmarks seldom proposed mitigation methods, with only a few focusing on co-occurrence or attention mechanisms to address these issues. in contrast, our paper analyzes token-level confidence at each layer to detect and promptly mitigate hallucinations. to handle the aforementioned research gaps, we present a specific definition for relation hallucination and propose the first comprehensive benchmark reefknot to evaluate the performance on relation hallucination. relation hallucination refers to the phenomenon in which mllms misinterpret the logical relationships between two or more objects in the corresponding image. unlike many benchmarks of mllm that were constructed by automatic labeling technique, we construct the relation-based dataset based on semantic triplets retrieved from the scene graph dataset. figure 2 also illustrates that relation hallucination can be more severe than object hallucination in current mllms, highlighting the importance of our evaluation. furthermore, we propose a simple relation hallucination mitigation method named detect-then-calibrate. this originates from the experimental observation that when relation hallucinations occur, the response probability drops significantly, hovering just above 50% in extreme cases compared to the usual nearly 90%. our method achieves an average improvement of 9.75% across three relation hallucination benchmarks. in summary, our main contributions are as follows: ## relation hallucination benchmark in this section, we describe the dataset construction pipeline of reefknot benchmark in figure 3 . unlike object and attribute hallucinations that only involve one entity, relations involve two entities making it more difficult to handle. we first identify relation triplets from visual genome (vg) dataset category y/n mcq vqa total #perception 5,440 4,800 2,150 13,260 #cognition 4,300 2,150 2,720 8,600 #total 9,740 6,950 4,870 21,880 ratio of positive and negative samples 1:1 number of perceptive relationship types 56 number of cognitive relationship types 152 number of images 11084 ## analysis of relation hallucination to quantitatively compare the decision probability distribution when hallucinations occur, we calculate the average probability of an equal number of relation hallucination and non-hallucination examples, as shown in note that because mllms have an extensive vocabulary, it becomes challenging to discern meaningful patterns in entropy variation when predicting the next word across the entire distribution. consequently, our analysis is restricted to observing the variation patterns of vocabulary within the range of potential answers x i . we present the ratio of hallucination cases to non-hallucination cases among both object and relation hallucination benchmarks in figure 6 . when e(x) > 0.6, relation hallucinations occur to a significant degree, indicating the effectiveness of our method to detect relation hallucination via entropy. to investigate the mechanism behind hallucination generation, we conducted an in-depth analysis of the fluctuations in probability values across model layers. mllms consist of a vision encoder, a projection layer, and a strong llm decoder, which is stacked by a set of n transformer layers, and an mlp layer ϕ(•) for predicting the next-word probability of distribution auto-regressively. given an image represented by vision encoder as v @xmath0, v 2 , . . . , v t } and a text prompt of tokens p @xmath1, p 2 , . . . , p t }, they are processed as a sequence h 0 = ψ(v t , p t ) through projection and concatenation function ψ(•). thus it can be observed that in the shallow layers (0th-20th), the probability of possible answers does not increase. we hypothesize that this is because, in shallow layers, the model is aggregating information to generate an answer. hallucination occurs in the deep layers. since deep layers contain a vast amount of knowledge, we speculate that they may cause relation hallucinations. ## detect-then-calibrate mitigation method the evaluation results indicate that mllms commonly suffer from severe relation hallucinations. as analyzed in section 4, we found that these relation hallucinations primarily stem from the model's lack of confidence. the lack of confidence in the model's responses results in a relatively high entropy value. therefore, we can detect the occurrence of relation hallucinations by monitoring changes in entropy. initially, we set a specific entropy threshold γ to detect potential hallucinations in the model's output. if the entropy of the model's response exceeds γ, which suggests a significant lack of confidence, we infer a high probability that the model has generated hallucinations. in cases where the model is identified to hallucinate, we will utilize the hidden states from intermediate layers to calibrate the final outputs layers, which is inspired by @xcite . note, unlike traditional contrastive decoding strategies @xcite @xcite , we do not calibrate all cases. instead, we focus on mitigating potential hallucination cases to avoid altering non-hallucinatory cases into hallucination ones. formula 5 show our calibration process. note that λ is the hyperparameter to control the degree of the intermediate layer; r is the token generated after calibration; α represents the degree of calibration operation. the algorithmic flow can be seen from the pseudo-code below: algorithm 1: detect-then-calibrate algorithm require: image vt; mllm m(•); prompt pt; uncertainty entropy threshold γ 1: h, r0, p, = m(xt, pt) 2: the entropy of generate token e(r0) = -n @xmath2: if e(r0) ≥ γ then 4: hallucination occurs! calibrate h by first λ layer 5: calibrated response @xmath3: else 8: normal response @xmath4: end if as illustrated in ## case study to intuitively demonstrate the generation process of hallucination, we visualized a real case from reefknot in figure 8 (nostalgebraist, 2020). in the image, when we ask "is the boy eating pizza in the photo?", mllms are unable to provide precise answers. to investigate this, we analyze the probabilities associated with the top five predicted tokens at each layer of the language model. as illustrated in figure 8 , the model typically converges on the correct answer as it progresses from shallow to deep layers. notably, hallucinations didn't occur until the final layer. additionally, we can observe that as the number of layers increases, the model increasingly focuses on narrowing down the range of answer choices, with the probability for both 'yes' and 'no' rising in deeper layers. however, as depicted in the changes of colors, when it reaches the final decoder block (layer 38-40th), the model's choices suddenly become uncertain and the entropy rises, accompanied by the emergence of hallucinations. so it is instinctive to utilize the entropy to detect hallucination and use logits of the intermediate layers, which are less likely to hallucinate, to calibrate the final logits to correct the answer. following our calibration utilizing equation 5, we not only dispel potential relation-level hallucination but also significantly bolster the confidence of model's responses. confidence calibration confidence estimation @xcite and calibration are essential for enhancing the reliability of llms such as @xcite . to assess the confidence associated with outputs from llms, @xcite have developed a method called semantic entropy that utilizes linguistic invariances reflecting shared meanings. however, this method relies on accessing token-level probabilities, which are often unavailable through current black-box apis. @xcite have designed prompts that encourage the models to selfassess their responses and to explicitly calculate the probability that an answer is true; while @xcite have prompted llms to provide both an answer and an accompanying confidence level. ## conclusion in conclusion, we propose a comprehensive benchmark called reefknot to evaluate and mitigate relation hallucinations in mllms. we construct the dataset with over 20k data through a scene graph based construction pipeline, covering two discriminative tasks (y/n and mcq) and one generative task (vqa). our in-depth evaluation highlights a substantial performance gap on relation hallucination in existing mllms, emphasizing the need for more sophisticated reasoning capabilities. subsequently, we discover that relation hallucinations tend to occur when mllms respond with low confidence. therefore, we propose a detectthen-calibrate method to mitigate the relation hallucination via entropy threshold, with an average reduction of 9.75% in the hallucination rate across reefknot and two other representative relation hallucination datasets. in general, we anticipate that the reefknot benchmark we propose will serve as a critical foundation for future developments in the field of trustworthy multimodal intelligence, enabling more reliable and robust systems.