Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Models (2025)

Abstract: Large Vision-Language Models (LVLMs) have demonstrated impressive performance on vision-language reasoning tasks. However, their potential for zero-shot fine-grained image classification, a challenging task requiring precise differentiation between visually similar categories, remains underexplored. We present a novel method that transforms zero-shot fine-grained image classification into a visual question-answering framework, leveraging LVLMs’ comprehensive understanding capabilities rather than relying on direct class name generation. We enhance model performance through a novel attention intervention technique. We also address a key limitation in existing datasets by developing more comprehensive and precise class description benchmarks. We validate the effectiveness of our method through extensive experimentation across multiple fine-grained image classification benchmarks. Our proposed method consistently outperforms the current state-of-the-art (SOTA) approach, demonstrating both the effectiveness of our method and the broader potential of LVLMs for zero-shot fine-grained classification tasks. Code and Datasets: https://github.com/Atabuzzaman/Fine-grained-classification

## introduction large vision-language models (lvlms) such as llava @xcite , blip-2 @xcite , instructblip @xcite , molmo @xcite , qwen2-vl @xcite , internvl @xcite , and deepseek's janus-pro @xcite and so on, alongside closed-source models like gpt-4v (ope-nai, 2024) and others have revolutionized multimodal understanding. these models demonstrate remarkable zero-shot capabilities in tasks that require extensive real-world knowledge, such as visual question answering (vqa), visual reasoning, lvlm lvlm with attention intervention brewer blackbird lvlm with attention intervention red-winged blackbird what is the name of the object in this image? which of the following best describes the object in the image? a. yellow-headed blackbird .............. e. brewer blackbird which of the following best describes the object in the image? a. yellow-headed blackbird .............. e. red-winged blackbird yellow-headed blackbird and image captioning. their success largely stems from instruction-tuning and pre-training on vast multimodal datasets, enabling effective integration of visual content with textual knowledge. however, despite their impressive performance in general vision-language tasks, lvlms face significant challenges in fine-grained image classification, which requires distinguishing subtle visual differences between highly similar categories, such as bird species or car models @xcite . unlike coarse-grained tasks where broad visual features suffice, fine-grained classification demands a precise understanding of nuanced visual characteristics and their alignment with detailed textual descriptions (figure 1 & 2). this challenge becomes particularly acute in zero-shot settings, where models must identify fine-grained categories without task-specific training. initial attempts to address this challenge include finer @xcite , which trained models to generate fine-grained attributes before predicting class names, and grain @xcite , which adapted clip @xcite with lvlms to measure similarity between generated text and ground truth class names. while these methods represent important first steps, their limited performance reveals critical gaps in current lvlm architectures and methodological approaches. although lvlms excel at generating descriptive captions and answering general questions about images, they struggle to capture and differentiate the subtle visual distinctions that define fine-grained categories @xcite . this limitation stems from their reliance on direct class name generation without sufficient domain knowledge and the use of less powerful models. these findings suggest that vision-language generation approaches may not adequately address the level of detail required for fine-grained classification tasks. to address these limitations, we propose a novel method for zero-shot fine-grained image classification using lvlms. inspired by vqascore @xcite , we first develop a strong foundation through likelihood-based yes/no qa classification. we then introduce an iterative multiple-choice question-answering (mcqa) approach to enable fine-grained classification. to reduce the computational overhead of iterative mcqa, we also explore an all-at-once approach that leverages the extended context capabilities of modern lvlms to process all class descriptions simultaneously in a single forward pass. rather than relying on direct class name generation, both approaches leverage the models' comprehensive visual understanding capabilities. our method aligns with traditional clip-based fine-grained classification approaches @xcite by utilizing class labels and descriptions. additionally, inspired by recent advances in lvlm hallucination reduction @xcite @xcite @xcite , we introduce a simple yet effective attention intervention technique. this intervention integrates insights from multiple studies, combining earlylayer visual information flow @xcite with deep-layer ground truth token detection capabilities @xcite to enhance classification accuracy. furthermore, we identify a critical limitation in existing class descriptions, which are typically generated using large language models (llms) such as gpt-3/4 @xcite or sourced from the internet and refined using gpt-4v @xcite . these descriptions often contain visual attributes that are absent from the actual class images or include irrelevant information that may impair model performance. to address this challenge, we develop curated class description benchmarks that more accurately capture the visual characteristics present in the images. our approach investigates settings both with and without explicit class names in the descriptions, providing greater flexibility and robustness in classification tasks. through extensive experiments on major fine-grained classification benchmark datasets, our methods demonstrate significant performance improvements over existing approaches and achieve sota performance. the main contributions of our work are as follows: ## proposed method this section presents our proposed method for zero-shot fine-grained image classification using lvlms. we begin by introducing a likelihoodbased yes/no question-answering (qa) approach, then propose an iterative mcqa framework to address the limitations of the initial approach. we introduce a simple yet effective attention intervention technique inspired by hallucination mitigation methods to enhance iterative mcqa approach performance. figure 2 provides an overview of our iterative mcqa method with attention intervention. finally, we discuss an all-at-once approach that leverages the extended context capabilities of modern lvlms to reduce computational overhead. ## experiments and evaluation to evaluate our proposed method, we conduct comprehensive experiments across five fine-grained image classification benchmark datasets using different lvlms and our curated class descriptions. this section presents the details of our class description curation process, experimental setup, evaluation results, and ablation studies. ## conclusion the application of large vision-language models (lvlms) to zero-shot fine-grained image classification represents a promising direction in visionlanguage understanding. we address current limitations by proposing multiple approaches: yes/no qa, iterative multiple-choice question answering (mcqa), and all-at-once methods, enhanced by a simple yet effective attention intervention technique. our investigation revealed the limitations of existing llm-generated class descriptions, prompting us to curate refined descriptions that better capture fine-grained visual characteristics. through comprehensive experiments across five datasets and multiple model architectures, we demonstrate that our iterative mcqa method consistently outperforms the yes/no qa approach, while the allat-once approach shows promising results for proprietary models but limitations for open-source architectures. our methods achieve substantial performance improvements, significantly outperforming existing sota approaches across all evaluated datasets. the effectiveness of our attention intervention mechanism, particularly its benefits for smaller models, suggests promising directions in developing more sophisticated attention guidance techniques tailored to specific lvlm architectures for zero-shot fine-grained classification tasks.