Game on Tree: Visual Hallucination Mitigation via Coarse-to-Fine View Tree and Game Theory (2024)

Abstract: Large Vision-Language Models (LVLMs) may produce outputs that are unfaithful to reality, also known as visual hallucinations (VH), which hinders their application in multimodal understanding and decision-making. In this work, we introduce a novel plug-and-play train-free decoding algorithm named Game and Tree based Hallucination Mitigation (GTHM), designed for mitigating VH. GTHM is inspired by empirical observations that the fuzziness of multi-granularity view perception exacerbates VH. Based on this, GTHM leverages visual information to construct a coarse-to-fine visual view tree (CFTree) that organizes visual objects, attributes, and relationships in a hierarchical manner. Additionally, we innovatively model the optimal visual-token matching process on the CFTree as the cooperative game. Specifically, we define the Tree-based Shapley Value (TSV) for each visual view on the CFTree to assess its significant contribution to the overall visual understanding, thereby determining the optimal visual granularity. Subsequently, we utilize the TSV as guidance to implement adaptive weight contrastive decoding to achieve vision-aware decoding. Extensive experiments on four popular benchmarks confirm the effectiveness of our GTHM in alleviating VH across different LVLM families without additional training or post-processing. Our code is published at https://github.com/mengchuang123/GTHM.

## introduction with the development of large language models (llms), large vision-language models (lvlms) have made significant progress in model architecture, training methods, and data diversity @xcite @xcite @xcite . lvlms excel at converting complex visual pat-prompt: please describe this image in detail ongoing response: on the beach, a man holding a ... ## preliminaries and motivation problem formulation. we consider a general lvlm, symbolized as θ, which is designed with an architecture that integrates a vision encoder, a vision-text interface, and a text decoder. initially, visual information v undergoes processing through the vision encoder to produce a visual embedding, which is then modified by the vision-text interface to align with the textual query x. the combined data serves as input to the text decoder, which autoregressively generates a textual output y as: figure 2 : analysis of the outputs of different lvlms on sampled 500 images in the mscoco validation set, consisting of (1) tsv distribution for different outputs of llava-1.5, (2) the relationship between sentence length and tsv, and (3) the relationship between prompt dependency measure (pdm) @xcite and tsv. ## experiments benchmarks. following common settings @xcite @xcite , we evaluate the effectiveness of our gthm in vh mitigation on four popular benchmarks: (1) quantitative metrics chair @xcite on mscoco dataset @xcite ; baselines. we adopt regular greedy decoding and beam search decoding methods, and various state-of-the-art (sota) decoding methods as baselines, including dola @xcite , opera @xcite , vcd @xcite , woodpecker @xcite , lure @xcite , and halc @xcite . we implement these baselines based on halc and compare them with our gthm under the same benchmarks and environment settings. backbones. following previous settings @xcite , we select popular lvlms llava-1.5 @xcite , minigpt-4 @xcite and mplug-owl2 @xcite as the backbones for all baselines except woodpecker and lure, where, woodpecker and lure utilize extra llms, i.e., @xcite and gpt-4 (achiam and steven adler, 2023), for self-correction and distillation. we investigate the hallucinations of these lvlms under different decoding and post-processing strategies to evaluate the effectiveness of our gthm. ## conclusions this work proposes a training-free, plug-and-play decoding strategy to mitigate vh in lvlms. our method was inspired by practical observations that ambiguity in multi-granularity view perception exacerbates vh. based on these, we construct the cftree to organize multi-granularity views. we innovatively model the retrieving optimal multigranularity views as a cooperative game. subsequently, we perform adaptive contrastive decoding based on game scores to achieve bias-free distribution. comprehensive experiments demonstrate the effectiveness of our gthm in reducing vh across different benchmarks and lvlm families.