Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models (2024)

Abstract: Fine-grained image classification, especially in zero-/few-shot scenarios, poses a considerable challenge for vision-language models (VLMs) like CLIP, which often struggle to differentiate between semantically similar classes due to insufficient supervision for fine-grained tasks. On the other hand, Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities in tasks like Visual Question Answering (VQA) but remain underexplored in the context of fine-grained image classification. This paper presents CascadeVLM, a novel framework that harnesses the complementary strengths of both CLIP-like and LVLMs VLMs to tackle these challenges. Using granular knowledge effectively in LVLMs and integrating a cascading approach, CascadeVLM dynamically allocates samples using an entropy threshold, balancing computational efficiency with classification accuracy. Experiments on multiple fine-grained datasets, particularly the Stanford Cars dataset, show that CascadeVLM outperforms existing models, achieving 92% accuracy. Our results highlight the potential of combining VLM and LVLM for robust, efficient and interpretable fine-grained image classification, offering new insights into their synergy.

## introduction the landscape of vision language models (vlms) has evolved rapidly, with models such as clip @xcite demonstrating remarkable zero-and few-shot classification capabilities @xcite . however, despite these advances, fine-grained image classification remains a significant challenge, particularly when distinguishing closely related subclasses @xcite . to address this, recent efforts have focused on improving clip's fine-grained classification performance through advanced prompt engineering @xcite and refinement of pre-training supervision @xcite ; singh et al., 2023). furthermore, @xcite introduced a method that uses gpt-3 to generate detailed class descriptions to improve the prompt context of clip. however, this approach struggles with visually similar classes, as the generated descriptions are often too similar, limiting its effectiveness in fine-grained classification tasks. in this paper, we explore large vision-language models (lvlms) to further harness their vast world knowledge for fine-grained classification. our initial experiments involved directly querying lvlms to classify images across multiple categories. although the overall accuracy was low, a critical insight emerged: when lvlms made correct predictions, the correct category frequently appeared early in the sequence of choices. moreover, we found that reducing the number of categories significantly improved accuracy, indicating that lvlms are susceptible to positional bias and face challenges with long-context modeling @xcite , as illustrated in figure 1 . building on these insights, we hypothesize that clip and lvlms have complementary strengths for fine-grained classification tasks. specifically, clip's contrastive pre-training allows it to score all possible categories for an image, which can be used to optimally order the categories for lvlms. this transforms lvlms' positional bias into an advantage. although clip's top-1 accuracy may be limited, its top-k accuracy is significantly higher, offering a broader set of correct options. for example, on the flowers102 dataset, clip (vit-b/32) achieves a 68.7% top-1 accuracy, which increases to 89.9% for top-10 accuracy. this characteristic motivates our approach to integrating clip with lvlms to enhance classification performance. we propose cascadevlm, a novel framework that combines the strengths of clip-like models and lvlms to achieve fine-grained image classification. the key innovation of cascadevlm is using clip-like models to filter and order class options, thereby enabling lvlms to perform more effectively. additionally, we leverage lvlms' incontext learning @xcite for few-shot tasks, and introduce an entropy-based threshold mechanism to improve inference efficiency by dynamically determining when to invoke lvlms, allowing for early exiting @xcite . our zero-and few-shot experiments on various fine-grained image datasets consistently demonstrate that cascadevlm outperforms standalone models. for instance, cascadevlm achieves 89.0% zero-shot and 92% few-shot accuracy on the stanford cars dataset. on the challenging inaturalist dataset, cascadevlm, which utilizes advanced clip-like models as a backbone in conjunction with lvlms, delivers superior performance. further analysis reveals that the primary performance gains stem from resolving uncertain and misclassified samples in clip predictions ( §4.1). additionally, we explore the sensitivity of classification results to class order ( §4.2) and examine the trade-offs between computational efficiency and performance with varying entropy thresholds ( §4.3). in summary, the key contributions of this paper are: (1) demonstrating the potential of lvlms for fine-grained image classification. (2) introducing the cascadevlm framework, which effectively integrates clip-like models and lvlms for zeroand few-shot fine-grained classification, providing new insights into their synergistic potential. ## analysis in this section, we explore various aspects of cas-cadevlm, highlighting the underlying reasons for its enhanced performance, the trade-off of the entropy threshold, and more. more investigations can be found in the appendix c. ## conclusion in this paper, we propose cascadevlm, harnessing the advantages of clip and lvlms for finegrained image classification. by utilizing clip for selecting the potential candidate class, lvlm can make more accurate predictions for image classes with subtle differences. experimental results on four benchmarks demonstrate the effectiveness of our proposed framework. further extension to the few-shot setups showcases the great potential of the cascading framework to leverage the in-context learning ability of lvlms. we experimented with various prompt designs to optimize qwen's performance in selecting the topk categories. two representative prompt styles were identified, each with distinct characteristics and performance implications. the first prompt style, while intuitive, occasionally led to non-compliant responses. for example, qwen would select a flower name not listed in the given options or use an alias instead of the specified name. this approach yielded suboptimal results. subsequently, we adapted our prompts to align more closely with the qwen training data, where the keyword "options" was prevalent. this adaptation significantly improved compliance and accuracy in the model's responses. thus, for the overall experiment, we use 'prompt2'. for gpt-4v, we applied a similar prompt style but followed the api requirement. prompt 1: picture 1: <img >.... @xmath0. ↩→ remember to select only one ↩→ flower name from the list , ↩→ and respond with the ↩→ flower name only . available ↩→ flower names : [...] prompt 2: picture 1: <img >... @xmath1: what is the flower ' s ↩→ name ? remember to select ↩→ only one flower name from ↩→ the options and respond ↩→ with the flower name only . ↩→ options : [...]