Mitigating Hallucinations in Large Vision-Language Models ( LVLM s) via Language-Contrastive Decoding ( LCD ) (2024)

Abstract: Large Vision-Language Models (LVLMs) are an extension of Large Language Models (LLMs) that facilitate processing both image and text inputs, expanding AI capabilities. However, LVLMs struggle with object hallucinations due to their reliance on text cues and learned object co-occurrence biases. While most research quantifies these hallucinations, mitigation strategies are still lacking. Our study introduces a Language Contrastive Decoding (LCD) algorithm that adjusts LVLM outputs based on LLM distribution confidence levels, effectively reducing object hallucinations. We demonstrate the advantages of LCD in leading LVLMs, showing up to %4 improvement in POPE F1 scores and up to %36 reduction in CHAIR scores on the COCO validation set, while also improving captioning quality scores. Our method effectively improves LVLMs without needing complex post-processing or retraining, and is easily applicable to different models. Our findings highlight the potential of further exploration of LVLM-specific decoding algorithms.

## introduction large vision-language models (lvlms) are a multimodal extension of large language models (llms), transforming textual prompts and image inputs into text. however, they frequently produce object hallucinations, where absent objects are mentioned in the output @xcite . while hallucination-mitigation techniques in llms are actively researched, specific strategies for lvlms are less developed. current methods involve model-specific adjustments, additional training, or auxiliary models for post-hoc correction, and are often proven inefficient, costly, or limited by training data and model biases @xcite @xcite according to the llm, the word "dog" is much more likely to appear next. lcd dynamically contrasts these probabilities to mitigate language biases in lvlm outputs. @xcite . conversely, lvlm hallucination evaluation has seen progress with object hallucination benchmarks like nope @xcite and pope @xcite , and recent works that aim for more holistic lvlm hallucination evaluation such as faithscore @xcite and hallusionbench @xcite . a key reason for lvlm hallucinations is their tendency to over-rely on linguistic information, as was first observed by @xcite . based on this insight, we propose to intervene in the lvlm decoding phase so that model outputs are less informed by language biases. specifically, we propose to use contrastive decoding @xcite to alter lvlm output probabilities with respect to the internal llm's probabilities, guided by a dynamic weighting mechanism based on the llm distribution's entropy. our experiments show that our proposed method, language contrastive decoding (lcd), improves hallucination scores on pope @xcite and chair @xcite on in-structblip variants based on vicuna and flan-t5 @xcite , llava 1.5 @xcite and mplug-owl2 @xcite . we asses lcd's overall generation quality by reporting captioning metrics and conducting a gpt4-v (openai et al., 2023) assisted evaluation. lcd, as a decoding strategy, can be applied to other models without additional training or output modifications, emphasizing its utility for broader lvlm use. the contributions of this paper are thus manifold. first, we introduce a novel decoding method tailored for lvlms to mitigate object hallucinations. next, we present a dynamic weighting strategy based on entropy which is applicable in various cd scenarios. finally, we share our code to encourage further research into lvlm-specific decoding strategies, a promising avenue for future research. ## motivation and background the integration of vision capabilities into llms has led to the development of large vision-language models, merging llms' textual understanding with vision-text encoders. this trend towards multimodal systems is exemplified in commercial platforms such as gpt4-v (openai et al., 2023) and google's gemini @xcite . ## language contrastive decoding (lcd) before presenting lcd, we briefly introduce the essentials of decoding in lvlms 3.1, followed by our formal proposal 3.2 and research hypothesis 3.3. ## experiments and results we set out to assess the effect of lcd on object hallucinations in lvlm outputs against popular decoding settings. additionally, we verify that lcd does not degrade output quality. to this end, we asses lcd on the pope benchmark (yifan li and wen, 2023), and on an image detailed-description task where we report hallucination and captioning metrics and conduct a gpt4-v assisted evaluation. polling-based object-probing evaluation pope consists of object-presence binary questions on 500 coco dataset images @xcite , with questions equally divided between present and absent objects. absent objects are chosen based on three criteria: random, popular (common in coco), and adversarial (commonly co-occurring with present objects). @xcite @xcite . ↑ means 'higher is better'. ↓ means 'lower is better'. ## results and discussion for the pope task, which evaluates object hallucinations using binary questions, lcd improves f1 scores across 11 out of 12 configurations compared to the baseline ( in the detailed-description task, which involves generating detailed descriptions of images, lcd significantly reduces hallucinations at both sentence and instance levels across all four models tested ( ## conclusion in this paper we present language contrastive decoding, a novel method to reduce hallucinations in lvlms. by dynamically adjusting output probabilities using the lvlm's internal llm, lcd significantly improves hallucination metrics across different lvlm architectures, enhancing the quality and reliability of generated content without necessitating retraining or auxiliary models and postprocessing. this work highlights the potential of specialized decoding strategies in enhancing multimodal ai models and lays the groundwork for further exploration into more sophisticated lvlm decoding methods.