Evaluating and Mitigating Object Hallucination in Large Vision-Language Models: Can They Still See Removed Objects? (2025)

Abstract: Large Vision-Language Models (LVLMs) have a significant issue with object hallucinations, where researchers have noted that LVLMs often mistakenly determine objects as present in images where they do not actually exist. Some recent studies evaluate the occurrence of object hallucinations by asking LVLMs whether they see objects that do not exist in input images. However, we observe that these evaluation methods have some limitations, such as the objects being questioned potentially having little relevance to the image. In this paper, we introduce a more challenging benchmark for evaluating object hallucinations by removing objects from images and then asking the model whether it can still see the removed objects. Our evaluation result reveals that LVLMs suffer from severe hallucinations, as they often still claim to see the removed objects. Through our analysis, we find that biases in training result in LVLMs lacking guidance on learning about the absence of objects, which in turn leads to a lack of ability to determine that objects do not exist in images. To address this issue, we further propose oDPO, a direct preference optimization objective based on visual objects. By guiding LVLMs to learn to determine the existence of objects, oDPO effectively alleviates object hallucinations. It achieves more competitive results than other hallucination mitigation approaches across multiple object hallucination benchmarks and enhances the performance of LVLMs in various vision-language tasks.

## introduction with the advancement of large language models (llms) @xcite @xcite and the emergence of powerful pre-trained vision-language models @xcite @xcite , several large vision-language ## the proposed rohe benchmark in this section, we introduce rohe ( §2.1) and the process of constructing the evaluation data ( §2.2). we then evaluate representative lvlms using rohe ( §2.3) and discuss the results ( §2.4). ## the proposed odpo approach to address the issue revealed in §2.4, we propose odpo, an object-based dpo objective designed to enhance lvlm' ability to determine the existence of objects, thereby mitigating hallucinations. ## conclusion in this paper, we introduce rohe, which designed to evaluate object hallucinations by removing objects from images. our evaluation results reveal that lvlms still suffer from severe hallucinations, as they often struggle to determine the absence of removed objects. to address this, we propose odpo, an object-based dpo objective designed to guide lvlms to learn to determine the existence of objects. we conducted extensive experiments and the results demonstrate that odpo not only enhances lvlms' ability to determine the existence of objects but also improves their performance on various vision-language tasks, particularly in reducing object hallucinations.