Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2023)

Abstract: Large-scale vision-language pre-trained (VLP) models are prone to hallucinate non-existent visual objects when generating text based on visual information. In this paper, we systematically study the object hallucination problem from three aspects. First, we examine recent state-of-the-art VLP models, showing that they still hallucinate frequently and models achieving better scores on standard metrics (e.g., CIDEr) could be more unfaithful. Second, we investigate how different types of image encoding in VLP influence hallucination, including region-based, grid-based, and patch-based. Surprisingly, we find that patch-based features perform the best and smaller patch resolution yields a non-trivial reduction in object hallucination. Third, we decouple various VLP objectives and demonstrate that token-level image-text alignment and controlled generation are crucial to reducing hallucination. Based on that, we propose a simple yet effective VLP loss named ObjMLM to further mitigate object hallucination. Results show that it reduces object hallucination by up to 17.4% when tested on two benchmarks (COCO Caption for in-domain and NoCaps for out-of-domain evaluation).

## introduction thanks to the advancement of large pre-trained language models (lms) and vision-language pre-training (vlp) methods, models are able to achieve surprisingly good performance in visionconditioned text generation, e.g., image captioning. however, large lms are found to generate unfaithful or nonsensical texts given the source input @xcite , which is called hallucination. the hallucination problem is also inherited by vlp models @xcite , as they are still lms that can understand visual information. vlp models often generate fluent and seem appropriate sentences if we only see the text, but wrong when taking the visual input into consideration. one major type of hallucination in vlp is known as object hallucination @xcite , where models generate texts containing non-existent or inaccurate objects from the input images. object hallucination in vlp models essentially limits their performance and raises safety concerns for industrial applications. for example, in biomedical image captioning @xcite , object hallucination reduces the accuracy of diagnosis and may lead to severe consequences for the patient. despite the limitations and risks caused by object hallucination, this problem has not been studied in contemporary vlp works yet. to narrow down the aforementioned research gap, in this paper, we systematically investigate four fundamental research questions about object hallucination: 1) how much do modern vlp models hallucinate? 2) how do different forms of image encoding affect object hallucination? 3) what are the effects of common vlp objectives on object hallucination? and 4) how to mitigate object hallucination in vlp models? for our first question, we examine recent stateof-the-art vlp models on the image captioning task. to evaluate object hallucination, we adopt and expand the chair (caption hallucination assessment with image relevance) metric proposed by @xcite . results show that these models still hallucinate frequently with ∼10% of the generated sentences containing at least one hallucinated object. this problem becomes much severer when generating sentences given out-ofdomain images. furthermore, we discover that the widely used optimization method scst @xcite could lead to more hallucination, even if it improves standard metrics like cider @xcite . for our second question, to investigate how different types of image encoding in vlp influence hallucination, we ablate three commonly used ones, including region-based, grid-based, and patch-based @xcite . surprisingly, we find that patch-based features perform the best and smaller patch resolution yields a non-trivial reduction in object hallucination. thirdly, we analyze the effects of commonly adopted vision-language pre-training objectives on object hallucination. specifically, we decouple and combine the image-text contrastive (itc) loss, the image-text matching (itm) loss with and without hard negatives, and the image-conditioned language modeling (iclm) loss. counter-intuitively, although itc and itm help to bring apart dissimilar images and texts, results show that they do not contribute much to alleviating object hallucination. the generative iclm loss is the main influential factor of object hallucination and different pre-training datasets lead to distinctive model behaviors. more detailed analysis is described in section 5.3. finally, we propose a simple yet effective new vision-language pre-training loss, namely objectmasked language modeling (objmlm), to further mitigate object hallucination by enhancing the alignment and restriction between text tokens and visual objects during generation. code and evaluation setups are released: https://github.com/ wenliangdai/vlp-object-hallucination . overall, our contributions are three-fold: ## evaluation setup in this section, we first introduce the chair evaluation metric and our proposed improvements to it in section 3.1. then, in section 3.2, we describe two datasets used for evaluation and explain how to calculate chair scores under such settings. ## object hallucination in vlp models benefitting from the vast advancement of various vlp methods, the performance of image captioning has been improved a lot by following a pretrainthen-finetune schema. generally, the performance is measured by metrics like cider in this section, we provide a thorough analysis of recent vlp models to investigate how much they hallucinate when generating text conditioned on visual information. the results are shown in overall, we observe two noteworthy insights. firstly, similar to the findings in @xcite , for all chair scores, they are not proportional to standard evaluation metrics. although standard metrics (e.g., the cosine similarity in cider) could potentially penalize the wrong object prediction, they do not directly reflect faithfulness. captions can still have good scores from standard metrics as long as they contain sufficient accurate objects to fulfill coverage, even if hallucinated objects exist. for example, vinvl large achieves higher cider and bleu-4 scores than vinvl base , but its chair scores are also higher. therefore, it is important to have a supplementary metric like chair to reflect faithfulness besides other metrics. secondly, the self-critical sequence training (scst) @xcite for the cider optimization method harms the faithfulness of generated captions. scst is a reinforcement learning algorithm that has been widely adopted as the secondstage finetuning after the standard cross-entropy optimization for image captioning @xcite @xcite @xcite . it calculates the reward based on the cider score by sampling captions during training without the need of another baseline. although scst can significantly boost performance on previous standard metrics, it encourages models to generate more hallucinated objects in the captions. for example, applying scst improves the cider score by 11.1 and bleu-4 score by 2.7 for vinvl base , yet it also increases 0.9 chair s score on the coco caption dataset. while @xcite also observed this phenomenon by testing small scale models, we show that scst hurts vlp models less. when the model is pre-trained very well, the side effect of scst is alleviated (e.g., the ofa large model). moreover, we demonstrate that this problem becomes more serious on out-of-domain images. for the vinvl base model, there are 10.9% more generated captions containing at least one hallucinated object after using scst. we speculate that the cider-based optimization encourages models to generate more words or phrases that have higher cosine similarities to the ground truth captions in the vision-language representation space, which can be plausible but not faithful. we show a case study in figure 1 . after finetuned by scst, models will take a bigger risk to generate more detailed yet incorrect information (e.g., in the second example in figure 1 , the sentence with hallucination generates the detailed information "mirror", which cannot be found in the image). this will further amplify the object hallucination problem on out-of-domain images, as models may have lower confidence in unfamiliar visual concepts. ## probing image encoding methods and vlp objectives in this section, we systematically study two determinants in vlp that are intuitively influential to the severity of the object hallucination problem. firstly, we study how different types of image encoding affect object hallucination, as they are the key components of models to interpret visual information. specifically, we ablate three encoding approaches including region-based, grid-based, and patch-based. secondly, we analyze how different vlp objectives influence object hallucination. we ablate three commonly used ones: image-text contrastive (itc), image-text matching (itm), and image-conditioned language modeling (iclm). implementation details are described in appendix a. ## object masked language modeling based on the findings in section 5, we propose a simple yet effective pre-training objective to mitigate object hallucination by improving object-level image-text alignment. it is named object masked language modeling (objmlm). as shown in figure 4 , objmlm can be seen as a variant of the mlm loss by masking all the objects in the text that appear in the image. for each sentence, we mask the object words and phrases as defined in the object category lists of both coco and no-caps by performing exact matching. similar to the whole word masking @xcite , we conduct whole object masking so that there will be only one [mask] token to replace each object. image encoder self attention vision-language cross attention feed forward bertlm ... linear a [mask] resting on the ground, as an [mask] looks on. a camel resting on the ground, as an antelope looks on. figure 4: an overview of the model architecture and the training of our proposed objmlm. we use the same architecture as described in section 5 to show the effectiveness of objmlm. here, the image encoder can be one of the region-based, grid-based, or patch-based variants as described in section 5.2. for objmlm, we use the vit-l/14. ## conclusion this paper systematically studies the objection hallucination phenomenon in vlp models, which is a severe problem but neglected in contemporary vlp works. we find that recent large vlp models still hallucinate frequently. moreover, the widely used scst method harms the faithfulness of generated sentences in image captioning, even if it improves previous standard metrics. furthermore, we discover that image encoding matters and the patch-based input with smaller resolution helps mitigate object hallucination. finally, we ablate commonly used vlp losses and show that token-level image-text alignment and controllability of the generation are crucial. we further propose a new loss named objmlm, which reduces object hallucination by 17.4% for an existing vlp setting. we believe our findings are beneficial for future work to build more responsible vlp models.