Mitigating Open-Vocabulary Caption Hallucinations (2024)

Abstract: While recent years have seen rapid progress in image-conditioned text generation, image captioning still suffers from the fundamental issue of hallucinations, namely, the generation of spurious details that cannot be inferred from the given image. Existing methods largely use closed-vocabulary object lists to mitigate or evaluate hallucinations in image captioning, ignoring the long-tailed nature of hallucinations that occur in practice. To this end, we propose a framework for addressing hallucinations in image captioning in the open-vocabulary setting. Our framework includes a new benchmark, OpenCHAIR, that leverages generative foundation models to evaluate open-vocabulary object hallucinations for image captioning, surpassing the popular and similarly-sized CHAIR benchmark in both diversity and accuracy. Furthermore, to mitigate open-vocabulary hallucinations without using a closed object list, we propose MOCHa, an approach harnessing advancements in reinforcement learning. Our multi-objective reward function explicitly targets the trade-off between fidelity and adequacy in generations without requiring any strong supervision. MOCHa improves a large variety of image captioning models, as captured by our OpenCHAIR benchmark and other existing metrics. We will release our code and models.

## introduction image captioning, the task of generating text that describes an image, is one of the most fundamental machine learning tasks combining vision and language. unfortunately, hallucinations plague the current state-of-the-art (sota), making it less usable for practical tasks that require confidence in the factual correctness of generated captions. consider, for instance, the image in figure 1 . sota image captioning models can generate text that is blip-2 a group of people jumping on a skateboard. ## the openchair benchmark to measure object hallucination in the openvocabulary setting, we propose the openchair (och) benchmark, consisting of ∼5k images illustrating diverse object types in context, accompanied by an evaluation procedure to measure object hallucinations in captioning models. following existing works @xcite @xcite , we consider our benchmark to be open-vocabulary as it contains diverse and uncommon items reflecting the unlimited distribution found in the real world, as well as having the ability to perform evaluation against arbitrary strings. openchair modifies the previous object hallucination metric chair @xcite , by relaxing its strong reliance on the object annotations in the ms-coco dataset, which constitute only 80 common object types. we control the diversity of object types in our benchmark by leveraging generative models to produce synthetic caption-image pairs, providing a complementary measure to chair's evaluation of a closed set of 80 common objects over natural images. the use of synthetic images for this purpose is further motivated by prior works which show that models training on synthetic image data may generalize to favorable performance on real images @xcite , as well as the recent growth in usage of synthetic data in general @xcite . we provide an overview of openchair below; further implementation details are provided in the appendix. in order to create a new benchmark that enables measuring the hallucination rate of arbitrary objects, while still maintaining high quality groundtruth captions, we use the pipeline illustrated in figure 2 . we first prompt the llm llama-2 (touvron et al., 2023) with few-shot examples of image captions from ms-coco, having it generate captions with a similar style but containing diverse details (and in particular, objects that are likely not contained in the closed set of ms-coco object labels). we then parse these synthetic captions with a syntactic parsing model, identify nouns with high concreteness scores @xcite (as these generally represent concrete objects), and balance the generated captions among object types to cover a wide array of objects. subsequently, we utilize the text-to-image diffusion model stable diffusion xl @xcite to generate images from these newly formed captions. this process results in a dataset that consists of synthetic images with corresponding captions including diverse, open-vocabulary objects. while this approach naturally scales to any number of desired image-caption pairs, we generate 5k such pairs (the same order of items found in the widely-used ms-coco karpathy test split) and perform manual filtering to assure each pair's alignment and general quality. in total, we removed a small minority (3%) of generated image-caption pairs. figure 3 shows examples of image-captions pairs from openchair. captioning models may predict free-text objects semantically matching the ground-truth while taking a different surface form (e.g. chihuahua vs. dog). to capture this in the open-vocabulary setting (rather than using a fixed list of synonyms as done in chair), we evaluate captioning models as follows: after predicting a caption for each image in the openchair dataset, we parse them to identify objects as described above. for each exchair openchair llm: man ∉ gt llm: guitar ∉ gt hallucinations: {man, guitar} man ≈ child guitar ∉ coco list hallucinations: {} prediction: a man playing the guitar figure 4: openchair vs. chair. in the above the predicted object guitar would not be counted by chair since it is not in its fixed vocabulary, while man would not be classified as a hallucination since it is defined by chair as a synonym of child. in contrast, open-chair's llm classifies both as hallucinations. ## the mocha framework to mitigate captioning hallucinations in the openvocabulary setting, we propose mocha, an rlbased pipeline using sota methods for stable reinforcement along with a carefully designed reward function that jointly optimizes for caption fidelity and semantic adequacy. figure 5 presents it. we turn to describe the learning procedure and objectives used in mocha. we start with preliminaries, then describe the reward function that mocha optimizes (section 3.1), and finally present the rl algorithm used for optimization (section 3.2). preliminaries. in general, rl views a model as an agent that interacts with the external environment and receives a reward, learning to optimize for this reward via exploring the environment (sutton and barto, 2018). in the case of image captioning, this model is a vlm operating in an environment of images and reference captions @xcite . during training, the agent generates a caption by sampling from its own predicted distribution as shown in figure 5 (left), receiving a reward based on an estimate of the caption quality. after collecting a full batch of rewards, a rl optimization step is applied as shown in figure 5 (right), and this process repeats iteratively until convergence. we use the following notation: let t and i be the sets of possible texts and images, with joint distribution x. given image @xmath0, an image captioning model m with weights θ induces a conditional probability distribution π θ (•|•) over generated captions ĉ ∈ t conditioned on images @xmath1. in the rl context, we refer to π θ as the policy. a reward function r : t × t × @xmath2; c, i) to generated caption ĉ relative to ground-truth caption c and image i. ## related work we provide a short summary of related works here, with an extended discussion of their methods and differences from our work in the appendix. measuring vlm hallucinations. several works have proposed holistic measures of generated text fidelity with respect to an input image using embedding similarities or learned metrics; such methods (the "similarity based" metrics of figure 10 ) include clipscore and variants @xcite , semantic fidelity @xcite , vifidel @xcite , and faier @xcite . while these metrics may correlate with the presence of hallucinations, they are less interpretable as they do not provide a discrete count of hallucinations in a predicted caption. by contrast, the pope metric ( li @xcite , and woodpecker @xcite . unlike these works, we mitigate hallucinations in the more challenging open-vocabulary setting. the contemporary work lure @xcite proposes a method for the open setting, but their proposed approach (complementary to ours) was not evaluated automatically in an open vocabulary setting due to the lack of an existing benchmark. figure 10 illustrates which explicit hallucination metric was used to evaluate each algorithm. as instruction-following vlms rapidly develop, multiple concurrent works have considered hallucinations in related tasks such as visual questionanswering (vqa), applying rl-based methods adopted from research on llms @xcite . these methods, which do not directly target our task, also require laborious human annotation to train a supervised reward model to penalize hallucinations, while our approach does not require any explicit supervision. deep rl for vlm text generation. deep rl has been widely applied to text generation tasks and specifically for optimizing classical imagecaptioning metrics (rennie et al., 2017; stefanini et al., 2022). another more recent development is the rise of deep rl for llms, which commonly uses the reinforcement learning from human feedback (rlhf) framework, which requires manual human preference annotation for training a reward model (ziegler et al., 2020; stiennon et al., 2020; ouyang et al., 2022). beyond llms, rlhf has been recently applied to aligning multimodal models with human preferences (abramson et al., 2022). while such methods succeed in optimizing sequence-level properties, they often suffer from increased hallucinations as a side-effect of optimizing for human preferences or standard nlg sequencelevel metrics (as illustrated in appendix c.5). ## conclusion we have shown the significance of operating in an open-vocabulary setting to effectively quantify and mitigate caption hallucinations. these are explicitly measured by our openchair benchmark, and our mocha framework allows for optimizing captioning models to reduce such hallucinations while preserving caption quality. this reduction is demonstrated on our benchmark and other existing metrics. our method and benchmark may be applied flexibly to a variety of model sizes and architectures, which we foresee providing a framework for future work on hallucination-aware captioning.