Mitigating Object Hallucinations in MLLM s via Multi-Frequency Perturbations (2025)

Abstract: Recently, multimodal large language models (MLLMs) have demonstrated remarkable performance in visual-language tasks. However, the authenticity of the responses generated by MLLMs is often compromised by object hallucinations. We identify that a key cause of these hallucinations is the model’s over-susceptibility to image frequency features in detecting objects. In this paper, we introduce Multi-Frequency Perturbations (MFP), a simple, cost-effective, and pluggable adversarial training method that leverages both low-frequency and high-frequency features of images to perturb visual feature representations and explicitly suppress redundant frequency-domain features during inference, thereby mitigating hallucinations. Experimental results demonstrate that our method significantly mitigates object hallucinations across various model architectures. Furthermore, as a training-time method, MFP can be combined with inference-time methods to achieve state-of-the-art performance on the CHAIR benchmark.

## introduction large language models (llms), exemplified by @xcite , demonstrate remarkable performance across diverse text-based tasks. by integrating visual encoders like clip @xcite , multimodal large language models (mllms) extend these capabilities to visual domains. these models excel in a range of visionlanguage tasks, including image caption @xcite , visual question answering @xcite , and visual dialogue @xcite . however, mllms always face the challenge of object hallucinations @xcite , where the model's outputs fail to accurately correspond to the objects in the real * equal contributions. to tackle this issue, previous studies propose various methods @xcite , which can be broadly categorized into training-time and inference-time methods. however, these approaches overlook a critical weakness in mllms. specifically, the mllms tend to be over-susceptible to frequency features in the image-namely, abrupt changes in pixel values-when detecting objects, often overlooking the actual features of real images. for instance, as illustrated in figure 1 , mllms can still identify objects in images, even when blurred images containing certain frequencies are used. this characteristic may contribute to the occurrence of object hallucination. this phenomenon has also been widely observed in studies on image frequency-domain attacks targeting vision models @xcite @xcite in other models. based on these insights, we introduce multi-frequency perturbations (mfp), a simple, costeffective, and pluggable method that leverages both low-frequency and high-frequency features of images to perturb visual feature representations and explicitly suppress redundant frequency-domain features during inference. mfp partitions an image into high-frequency and low-frequency components, extracts their respective features, and performs a fine-grained fusion of these features with the original image features at the visual token level. this process results in more robust image features. the module is composed of two main components: (1) multi-frequency feature extraction. in this step, gaussian high-pass and low-pass filters @xcite are applied to the original image to obtain its raw high-frequency and low-frequency features. these raw features are then fed into the visual encoder, producing the corresponding high-frequency and low-frequency the image shows a plate with two large pieces of broccoli covered in melted cheese. the cheese is browned and bubbly, indicating it has been baked or broiled. the broccoli is vibrant green, and the cheese appears creamy and slightly golden on top. in the background, there are a few pieces of roasted or fried potato wedges with a browned, crispy texture. the dish is presented on a white plate, giving it a clean and appetizing appearance. ## over-susceptibility to frequency in mllm unlike most previous work that attributes hallucinations to text tokens' interference or insufficient attention to visual tokens, we find that the models are over-susceptible to the frequency of the image (that is, how drastically the pixel values of the image change) when recognizing objects. we explore two experimental settings: (1) natural images and ## multi-frequency perturbutaions in section 3, we demonstrate that mllms are overly sensitive to frequency features during object recognition, which leads to object hallucinations. in this section, we present our proposed robust training method based on multi-frequency perturbations to address this issue. model architecture our model architecture, similar to most mllms, as shown in figure 5 , ## conclusion we present the first study of object hallucination in mllms from a frequency-domain perspective, showing that models are overly sensitive to frequency features during object recognition. to address this, we propose mfp-a simple and efficient training-time method that effectively mitigates hallucinations. mfp achieves strong results across multiple benchmarks, generalizes well across architectures, and can be seamlessly combined with inference-time methods like pai to achieve state-of-the-art performance on the chair benchmark. these findings underscore the potential of frequency-domain techniques for hallucination mitigation and highlight the importance of further investigating the relationship between image features and object recognition in mllms.