Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models? (2024)

Abstract: Large vision-language models (LVLMs) have recently dramatically pushed the state of the art in image captioning and many image understanding tasks (e.g., visual question answering). LVLMs, however, often hallucinate and produce captions that mention concepts that cannot be found in the image. These hallucinations erode the trustworthiness of LVLMs and are arguably among the main obstacles to their ubiquitous adoption. Recent work suggests that addition of grounding objectives—those that explicitly align image regions or objects to text spans—reduces the amount of LVLM hallucination. Although intuitive, this claim is not empirically justified as the reduction effects have been established, we argue, with flawed evaluation protocols that (i) rely on data (i.e., MSCOCO) that has been extensively used in LVLM training and (ii) measure hallucination via question answering rather than open-ended caption generation.In this work, in contrast, we offer the first systematic analysis of the effect of fine-grained object grounding on LVLM hallucination under an evaluation protocol that more realistically captures LVLM hallucination in open generation. Our extensive experiments over three backbone LLMs reveal that grounding objectives have little to no effect on object hallucination in open caption generation.

## introduction large vision-language models (lvlms) have recently displayed impressive image understanding abilities @xcite @xcite @xcite . their widespread adoption, however, is hindered by object hallucination in which the lvlms-similar to "general" hallucination of llms @xcite )-"invent" objects (or attributes of or relations between objects) not present in the image. a range of methods have recently been proposed to address lvlm hallucination such as modified decoding strategies @xcite , post-hoc removal of hallucinated content @xcite , or reinforcement learning @xcite @xcite . most of these approaches, however, either increase inference cost or need expensive additional training and/or data, impeding their ubiquitous applicability. a recent line of work @xcite @xcite has suggested that including grounding objectives-e.g., based on referring expressions @xcite where textual descriptions of image regions have to be grounded to the respective parts of the imageinto the lvlm training reduces object hallucination. the claim is intuitive: region-level objectives demand finer-grained image understanding than the 'global' image captioning (de facto the main training objective of lvlms), as demonstrated in visiolinguistic compositionality @xcite . such objectives should thus, intuitively, discourage models from generating content they cannot ground in the image. intuition aside, the empirical support for the claim that grounding objectives reduce lvlm hallucination is weak and mainly limited to question-answering (qa) style of evaluation in which the model is explicitly asked about existence of objects in an image @xcite ; we argue that this evaluation protocol poorly aligns with real-world free-form text generation tasksprimarily open image captioning-for which there is no empirical evidence yet that object grounding reduces hallucination. contributions. in this work, we perform the first comprehensive analysis of the effects that grounding objectives have on lvlm object hallucination in open (i.e., free-form) image captioning, addressing the shortcomings of existing hallucination evaluation protocols. concretely, we measure the effect of adding two popular grounding objectives as additional objectives to standard image captioning-based training of lvlms: (1) the referring expressions (re) objective asks the model to generate the bounding box of the region that corresponds to a textual description and vice versa; whereas (2) the grounded captioning (gc) objective demands that the model generates image descriptions with interleaved (relative coordinates of) bounding boxes for mentioned objects. we then compare the extent of hallucination for lvlm variants trained with and without these grounding objectives. to this end, we compare the hallucination measures based on question answering (qa) @xcite against free-form metrics for open captioning @xcite . critically, observing that (1) existing evaluation measures and protocols @xcite rely on mscoco @xcite and (2) mscoco data is part of the training mix for most lvlms, we argue that existing measures are likely to underestimate lvlm hallucinate; we thus extend our hallucination evaluation protocol to out-of-distribution data that lvlms will not have seen in training. findings. our experiments with three different llm backbones show that, under a sound evaluation protocol, including grounding objectivesreferring expressions and grounded captioning-to lvlm training has little to no effect on object hallucination, both in qa-based evaluation and open-ended captioning. enforcing generation of grounded captions at inference time, on the other hand, slightly reduces object hallucinations but the effect is small and comes at the cost of (slight) reduction in caption detailedness. a qualitative inspection of grounded captions also confirms that forcing model to generate a bounding box for mentioned objects most often does not prevent it from hallucinating content. in sum, we find that grounding objectives fail to meaningfully reduce lvlm hallucination, calling for novel methodological proposals towards hallucination reduction. ## grounding objectives in lvlms grounding objectives seek to align natural language expressions with regions in the image. these objectives either take image regions as input, in the form of a bounding box and predict corresponding language expressions or produce such regions as output. many recent lvlms have been trained with grounding tasks in their training mix alongside standard tasks like captioning and vqa @xcite @xcite ; other models have been designed specifically for expression grounding and trained with grounding objectives only @xcite @xcite @xcite @xcite . objectives. our investigation focuses on the two arguably most popular grounding objectives, commonly part of lvlm training: referring expressions @xcite and grounded captioning @xcite . referring expressions is the standard grounding objective, included in training of nearly all lvlms. given a natural language description (of a region), the model has to ground it to the correct image region. as is common practice, we also use the inverse task, that is, generation of the natural language description for the given image region. grounded captioning is the task of generating an image caption in which the locations of regions for mentioned objects are interleaved in the caption (see figure 2 for examples). in theory, such explicit grounding is expected to result in closer adherence to the image content and reduce hallucinations. other grounding objectives have been proposed for lvlms training, such as question answering with image regions in the input or output @xcite ; these, however, are outside the scope of our study, because we focus on the effects of grounding on hallucination primarily in free-form captioning. ## measuring object hallucination lvlm object hallucination is evaluated via two main protocols: (1) in qa-based evaluation, where models answer questions about object existence in the image @xcite and (2) in open gener-chair 1. 2. 3. 1.there is a hound. 2.there is a cat. 3.the hound is white. faithscore dog cat collar human 1. match to set 1. llm extracts facts. 2. vqa verifies facts 2. validate with human annotation chair relies on human object annotation (over a fixed set) to identify objects and check if they are hallucinated. faithscore first uses an llm to convert captions into facts which are then verified by a vqa model. qa-based hallucination evaluation. pope @xcite is the de facto standard benchmark for qa-based hallucination evaluation. relying on images annotated with objects from mscoco @xcite , the benchmark consists of yes/no questions about object existence ("is there x in the image?"). the negative questions-about objects not in the image-are generated in three different ways using: i) objects randomly selected from the total pool of objects that exist in the dataset (random); ii) the most frequently annotated objects in the dataset (popular); iii) objects with high co-occurrence to the image's actual objects (adversarial), as co-occurrence statistics are a common cause of hallucinations @xcite @xcite . the performance metric is accuracy, i.e., the percentage of correctly answered questions. open hallucination evaluation. we focus on two popular meatrics for quantifying hallucination in open caption generation: chair @xcite and faithscore @xcite , illustrated in figure 1 ). the two metrics identify hallucination in different ways: by complementing them with one another, we mitigate the risk of our findings merely being an artifact of a single (imperfect) evaluation metric. both metrics can also indirectly quantify how informative and descriptive the generated captions are. as our result will show ( §5), there exists a tradeoff between faithfulness/hallucination and informativeness of the captions. we thus argue that the hallucination metrics should be contextualized with the measures of informativeness: factually correct but uninformative captions are as undesired as captions with hallucinated information. chair detects hallucinated objects using the set of 80 object classes from mscoco @xcite with which the images are annotated. words from the captions are matched-using exact string matching-against the class names, augmented with synonyms. the resulting list of matched objects is then cross-referenced against the gold list of annotated objects and all matched but not annotated objects are considered hallucinations. two scores are produced over the dataset: (1) chair i divides the total number of hallucinated objects across all captions with the total number of detected objects; (2) chair s is the proportion of images in the dataset for which the caption contains at least one object hallucination. chair s is less than ideal for longer captions as they are more likely to contain at least one hallucination; such a binary caption-level measure would hide potentially substantial differences in hallucination rates between models. because of this, we adopt only chair i in this work. following @xcite , we additionally report the average number of matched objects per caption as well as the gold object coverage (i.e., the average percentage of annotated objects mentioned in the caption) as measures of caption informativeness. chair unfortunately comes with two major shortcomings. first, it is based on mscoco images and object annotations which are widely used in a range of derivative datasets leveraged for training lvlms @xcite @xcite . this makes lvlms a priori less likely to hallucinate on mscoco images, which means that chair is likely overly optimistic about (i.e., it underestimates) the amount of lvlm hallucination "in the wild". we thus propose to extend chair to an out-of-distribution dataset, one that ideally also comes with a larger set of object classes. second, chair relies on exact string matching between caption words and synonym sets of the object classes. adapting vanilla chair based on string matching to a larger set of object classes would, however, require significant manual effort, as one would have to (1) create a curated list of synonyms for all new classes (without overlap between related classes) to correctly account for recall and ( 2 ) inspect examples and create special rules for edge cases to limit false positives (e.g., add 'baby x' synonyms to all animal classes 'x' in order not to falsely match the 'person' class). addressing both issues simultaneously, we propose semantic matching between the caption and object classes as an alternative to string matching for large sets of object classes. our extension, dubbed chair-men (from chair with matching using embeddings of noun phrases) (1) extracts all noun phrases from the generation, 1 (2) embeds the extracted phrases as well as classes names with a pretrained sentence encoder (reimers and gurevych, 2019) 2 and (3) makes matching decisions based on cosine similarity between obtained embeddings: to each noun phrase, we assign (i) the class amongst the image's objects with the most similar embedding, if cosine exceeds a threshold t 1 , (ii) the class amongst the other objects (i.e., not present in the image) with the most similar embedding, if cosine exceeds a threshold t 2 , or otherwise (iii) no object class. matching first only against the image's objects makes false negatives from a semantically related object not in the image less likely. we calibrate the thresholds (t 1 = 0.73, t 2 = 0.78) by trying to match the scores that vanilla chair produces on mscoco, as an established measure for that dataset. faithscore @xcite , a model-based hallucination metric, is designed with finer-grained evaluation in mind: it does not only consider objects/entities but also other aspects that models can hallucinate about (specifically: color, relation, count, and 'other' attributes), without the need for human annotation. faithscore computation is a 2stage process that relies: (1) on an llm to extract 'atomic facts' from the generated text, phrasing 1 with spacy v3 en_core_web_sm 2 baai/bge-base-en-v1.5 @xcite them as statements (e.g., "there is a man") the factuality of which, in the context of the image, is then (2) verified with a vqa model (question: "is the following statement correct?"). the final score is then simply the proportion of positive answers given by the vqa model. we additionally report the average number of facts produced by the llm as a measure of informativeness of generated captions. the original work of @xcite relies on gpt-4 to extract facts but this is too expensive for our evaluation; instead, we use a smaller llm foot_0 after verifying that it successfully follows task instructions. we use ofa @xcite as the vqa model for faithscore, as it is much faster and only marginally less accurate than llava-1.5 @xcite according to @xcite . ## experimental setup we comprehensively analyze the effect of grounding objectives on lvlm hallucination. for the sake of transferability and robustness of our findings, our experimental core, namely the model architecture and training procedure, follows established practices as closely as possible. all model instances are trained according to the same protocol, that is, we control for everything other than the effect of grounding, i.e., inclusion/exclusion of grounding data during training. we primarily focus on measuring hallucination in open-ended image captioning as this, we argue, better reflects lvlm's hallucination in real-world applications; for completeness and comparison of evaluation protocols, we also perform the qa-based evaluation with pope. we benchmark lvlms for hallucinations in two different caption generation scenarios: (1) in standard image captioning, with expected caption length of 1-2 sentences (as in mscoco), and (2) grounded image captioning (with standard length), where the lvlm is explicitly prompted to interleave region coordinates into the caption. in the appendix b, we also provide results for long (i.e., detailed, descriptive) caption generation. evaluation datasets. despite the previously mentioned shortcomings, mscoco @xcite remains the primary dataset for evaluating lvlm hallucination in the literature, both with qabased and free-form generation metrics/protocols @xcite . we thus include mscoco but complement it with the objects365 (o365) @xcite dataset which comes with a much larger inventory of object classes (365 classes in total, including the 80 mscoco classes) and, consequently, more object annotations per image. we evaluate on 5000 and 5386 images from test portion of mscoco and validation portion of o365, respectively. 5 for the pope evaluation, we generate two new test sets from o365, each with 1500 examples (matching mscoco pope): o365/coco uses only the 80 classes from mscoco, and o365/non-coco utilizes the remaining 285 classes. ## results we now report the observed hallucination effects under both protocols: in free-form captioning and in qa-based hallucination evaluation (as indicated by the pope metric/protocol). the reported chair results correspond to our chair-men variant; we report the results obtained with the vanilla chair based on string matching in appendix c. we did not separately optimize hyperparameters for each llm and will thus refrain from their mutual performance comparison; instead, for each of the three llms, we analyze how inclusion of grounding objectives affects their hallucination. referring expressions. before we test the effects of grounding on free-form and qa-based hallucination, we first analyze if the two grounding objectives are mutually compatible. concretely, we test how the models trained with grounding objectives (+re, and +re+gc) perform on one of the grounding tasks itself. in other words, we test if and how well models explicitly trained with grounding objectives learn to ground expressions and whether the two grounding objectives are mutually beneficial. the results for expression grounding (one of the two re tasks: given the description, provide the bounding box) are shown in qa hallucinations with pope. ## qualitative grounded caption analysis we show examples for grounded captioning in figure 2. the grounding itself does not necessarily prevent the model from hallucinating: in the first standard: a painting of a woman with a vase and oranges. grounded: an artistic painting of a woman with a vase . standard: two elephants are in a field near water. grounded: two elephants are in a field with water. standard: a small bird is standing in a pot of food. grounded: a black bird is eating a peeled apple out of a pot . example, the model fully hallucinates a woman along with a bounding box for her. in the second example, the second 'elephant' bounding box is positionally correct in that it points to an animal, but that animal is a rhino. in the third example, similarly, the bounding box correctly contains an apple but the attribute 'peeled' is hallucinated. these ## related work large vision-language models. lvlms are essentially large language models (llms) @xcite @xcite extended to "understand" visual input. recent models have shown an impressive understanding of images @xcite @xcite @xcite @xcite @xcite @xcite and a range of models have been proposed specifically for grounding and referring @xcite @xcite @xcite @xcite . ## conclusion object hallucination remains one of the main obstacles to wide-range adoption of lvlms. prior work suggested that grounding objectives like referring expressions reduce hallucination but the empirical support for this claim is confined to qa-based evaluation. in this work, we carried out an in-depth analysis of the effects that grounding objectives in lvlm training have on their hallucination in open image captioning. our extensive experiments with three backbone llms show that there is no causal link between improved object grounding (via objectives like referring expressions) and hallucination reduction: this observation is true both under qa-based and open captioning hallucination evaluation protocols. finally, we observe that explicitly prompting lvlms to generate grounded captions at inference can slightly reduce hallucination but at the expense of reduced caption informativeness.