Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models (2024)

Abstract: Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease. While such capability is largely attributed to the rich world knowledge contained within the Large Language Models (LLMs), our work reveals their shortcomings in fine-grained visual categorization (FGVC) across six different benchmark settings. Most recent state-of-the-art LVLMs such as LLaVa-1.5, InstructBLIP and GPT-4V not only severely deteriorate in terms of classification performance, e.g., average drop of 65.58 in EM for Stanford Dogs for LLaVA-1.5, but also struggle to generate descriptive visual attributes based on a concept that appears within an input image despite their prominent zero-shot image captioning ability. In-depth analyses show that instruction-tuned LVLMs suffer from modality gap, showing discrepancy when given textual and visual inputs that correspond to the same concept. In an effort to further the community’s endeavor in this direction, we propose a multiple granularity attribute-centric benchmark and training mixture, Finer, which aims to establish a ground to evaluate LVLMs’ fine-grained visual comprehension ability and provide significantly improved explainability.

## introduction in recent years, large vision-language models (lvlms) that are able to generate image-grounded text have seen significant progress. models such as instructblip @xcite and llava @xcite have consistently exhibited strong zero-shot capability in generating image captions, visual reasoning and textual descriptions, and even leveraging external knowledge for complex question answering tasks @xcite . such results across diverse benchmarks indicate that these models, most of them being built on large language models (llms) zero-shot downstream tasks input image ## fine-grained image understanding in vision-language models we first evaluate the fine-grained visual categorization performance of five different instruction-tuned baselines on six different fgvc benchmarks. this section elaborates on the details of the experimental setup and models investigated in this work. ## modality gap: discrepancy between textual and visual modalities we hypothesize that the lack of zero-shot concept classification ability of lvlms arises mainly due to the modality gap between textual and visual inputs, preventing the models from leveraging the existing concept-related parametric knowledge when an image of a concept is given. note that modality gap studied in this work is different from the one previously identified in clip-like vlms @xcite . in this section, we aim to delve into the details of how these models process visual and textual modalities by exploring how well they perform when given only textual descriptions of a concept ( §4.1) and how they accurately elaborate what they see in a given image ( §4.2). we also perform linear probing ( §4.3) against projected and original vision encoder output embeddings to gauge the influence of projection and the subsequent loss of visual information on the modality gap. ## finer: fine-grained concept recognition benchmark to facilitate research for fine-grained image understanding in lvlms, we propose a new benchmark and training mixture, finer. finer intends to evaluate the interplay between the concept image understanding and attribute knowledge, in addition to the training mixture that mitigates the modality gap and enhances fine-grained concept recognition.