Making LVLM s Look Twice: Contrastive Decoding with Contrast Images (2025)

Abstract: Large Vision-Language Models (LVLMs) are becoming increasingly popular for text-vision tasks requiring cross-modal reasoning, but often struggle with fine-grained visual discrimination. This limitation is evident in recent benchmarks like NaturalBench and D3, where closed models such as GPT-4o achieve only 39.6%, and open-source models perform below random chance (25%). We introduce Contrastive decoding with Contrast Images (CoCI), which adjusts LVLM outputs by contrasting them against outputs for similar images (Contrast Images - CIs). CoCI demonstrates strong performance across three distinct supervision regimes. First, when using naturally occurring CIs in benchmarks with curated image pairs, we achieve improvements of up to 98.9% on NaturalBench, 69.5% on D3, and 37.6% on MMVP. Second, for scenarios with modest training data (~5k samples), we show that a lightweight neural classifier can effectively select CIs from similar images at inference time, improving NaturalBench performance by up to 36.8%. Third, for scenarios with no training data, we develop a caption-matching technique that selects CIs by comparing LVLM-generated descriptions of candidate images. Notably, on VQAv2, our method improves VQA performance even in pointwise evaluation settings without explicit contrast images. Our approach demonstrates the potential for enhancing LVLMs at inference time through different CI selection approaches, each suited to different data availability scenarios.

## introduction large vision-language models (lvlms) are becoming increasingly popular for text-vision tasks that require reasoning over both modalities. however, they often struggle with fine-grained visual discrimination -that is, the ability to tell two similar yet distinct images apart -a crucial capability for real-world applications such as multimodal search, manufacturing, and robotics. recent benchmarks have exposed this limitation: on naturalbench @xcite , which tests visual question answering over closely related images, state-of-the-art closed models like @xcite achieve only 39.6% accuracy. similarly, on the d3 benchmark @xcite , which requires describing differences between paired images, open-source models perform below random chance (25%). efforts to address fine-grained visual discrimination in lvlms are still under-explored. current strategies addressing other lvlm shortcomings often rely on fine-tuning with specialized datasets @xcite @xcite , multi-step correction pipelines @xcite , or inference-time methods @xcite @xcite . inference-time methods are particularly appealing as they do not require expensive model training and are less prone to compounding errors that can affect multi-step systems. building on the advantages of inference-time methods, we propose contrastive decoding with contrast images (coci), an approach specifically designed to improve fine-grained visual discrimination in lvlms. coci penalizes lvlm next-token probabilities with those obtained by feeding a different, contrasting image input (see figure 1 ). we evaluate coci across three different supervision regimes. first, using naturally occurring contrast images in curated benchmarks like nat-uralbench, d3 and mmvp, we demonstrate improvements up to 98.9%, 69.5%, 37.6% respectively. this establishes a performance ceiling for coci when ideal cis are available. for applications where natural cis are unavailable but training data exists, we show that a lightweight classifier can effectively select cis from visually similar images at inference time, improving naturalbench performance by up to 36.5%. in settings without training data, we propose a caption-matching technique that selects cis at inference time by comparing lvlmgenerated descriptions of candidate images. experiments with leading lvlms -qwen2-vl, llava-onevision, and llama 3.2 @xcite @xcite -establish the potential of contrastive decoding strategies with contrastive images for improved multimodal reasoning in real-world tasks. ## contrastive decoding with contrast images (coci) we present coci, a method to improve lvlm outputs by penalizing token probabilities that are likely under a contrast image. the choice of contrast image is crucial: e.g., when querying about fruit ripeness with an input image of an unripe banana, contrasting against an image of a ripe banana provides strong contrastive signal, while an image of a ripe pear offers weaker contrast and an image of a bus provides no useful signal and may degrade performance. this intuition guides our ci selection strategies across different scenarios. before formalizing this intuition, we first review key concepts in lvlm text generation. ## experiments we evaluate coci using three leading lvlms 7 on four benchmarks, three specifically targeting finegrained visual discrimination: naturalbench @xcite evaluates similar image discrimination through yes/no and multiplechoice questions, with different answers for paired images. the benchmark contains 1900 image pairs (two questions per pair), split into train (60%), dev (20%), and test (20%) sets. we measure image accuracy (both questions correct), question accuracy (per-question), and group accuracy (all four image-question combinations correct). @xcite evaluates visual difference detection through multiple-choice questions on 150 image pairs. each pair differs in specific visual aspects (object state, position, or orientation). success requires correct answers for both images in a pair. d3 (detect, describe, discriminate) @xcite assesses models' ability to generate discriminative descriptions between similar images across 247 pairs. we adapt d3 for coci by treating it as a single-input task, generating separate descriptions per image. evaluation follows the original self-retrieval protocol, measuring whether an image-text encoder correctly matches descriptions to their images. ## results and discussion in in naturalbench, g-acc shows particularly strong improvement with natural cis as it requires consistency across all image-question combinations. this pattern persists with classifier-selected cis, where g-acc improves by up to 10.2 points while other metrics show modest gains. the substantial gap between natural cis and other methods suggests that classifier-selected and captionmatched cis, while beneficial, don't yet capture all aspects that make natural pairs effective. foot_5 5 related work inference-time methods for enhancing multimodal reasoning. recent work has focused on hallucination reduction through confidence-based adjustments @xcite , semantic references @xcite , and contrastive decoding with perturbed inputs @xcite . our work extends these approaches to fine-grained visual discrimination. ## conclusion we introduced contrastive decoding with contrast images (coci), demonstrating its effectiveness in improving lvlms' fine-grained visual discrimination capabilities in both vqa and longform generation tasks. while naturally occurring contrast pairs yielded the strongest gains, both classifier-based and caption-matching approaches provide meaningful improvements without requiring dataset curation or model training. we validated the generality of our method through experiments with caption-based contrast selection, showing that coci does not rely on pre-curated pairs but can leverage them when available. notably, coci improves performance even on tasks that don't explicitly measure fine-grained discrimination. our results show that contrastive decoding algorithms, when combined with strategic contrast image selection, improve lvlms' ability to make fine-grained distinctions and their overall vqa abilities, opening new avenues for improving multimodal reasoning through inference-time techniques.