A frican or E uropean Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification (2024)

Abstract: Recent Large Vision-Language Models (LVLMs) demonstrate impressive abilities on numerous image understanding and reasoning tasks. The task of fine-grained object classification (e.g., distinction between animal species ), however, has been probed insufficiently, despite its downstream importance. We fill this evaluation gap by creating FOCI ( F ine-grained O bject C lass I fication), a difficult multiple-choice benchmark for fine-grained object classification, from existing object classification datasets: (1) multiple-choice avoids ambiguous answers associated with casting classification as open-ended QA task; (2) we retain classification difficulty by mining negative labels with a CLIP model. FOCI complements five popular classification datasets with four domain-specific subsets from ImageNet-21k. We benchmark 12 public LVLMs on and show that it tests for a complementary skill to established image understanding and reasoning benchmarks. Crucially, CLIP models exhibit dramatically better performance than LVLMs. Since the image encoders of LVLMs come from these CLIP models, this points to inadequate alignment for fine-grained object distinction between the encoder and the LLM and warrants (pre)training data with more fine-grained annotation. We release our code at ANONYMIZED .

## introduction large vision language models (lvlms)-large language model (llm) that have been adapted to process images as input alongside text-have shown impressive performance on a wide range vision-language tasks @xcite @xcite . lvlms are mutually compared using a range of benchmarks that test for various image understanding and reasoning skills, such as existence and counting of objects, localization, comparison between objects name this dog breed and tell me a fact about it. llava 1.5: the dog in the image is a samoyed, which is a breed of dog known for its thick, white double coat. [...] idefics-2: keeshond. these dogs were originally bred as ratters and watch dogs in holland. figure 1 : the importance of object recognition: llava 1.5 fails to identify the dog breed. idefics-2 correctly recognizes it and gives a correct fact as a result. lvlms are, however, barely ever tested for finegrained object classification-the ability to correctly recognize different animals, plants, or manmade objects-which is, we argue, an important skill that complements general image understanding. 1 besides it being an end-task in itself, e.g., to answer questions such as "what is this flower / animal / car / building / ... called?", it is often implicitly needed in information-seeking situations, where the success depends on the models' ability to correctly and precisely identify an object ("is this mushroom edible?", "how to cook this dish?", or "what is this dog breed used for?") @xcite @xcite . as illustrated in figure 1 , only one of the lvlms correctly identifies the dog breed in the image and can follow up with relevant information. note that this is different from general l(v)lm hallucination @xcite , where models 'invent' incorrect information. instead, the generated content is correct for the object, but the object is misclassified: the information about samoyed by llava 1.5 is correct, but the dog in the image is a keeshond. to fill this gap in lvlm evaluation, we create a comprehensive benchmark dubbed foci (finegrained object classification) that tests models' fine-grained object recognition over a wide range of object categories. our key contribution is a welldefined task formulation that avoids pitfalls of prior work: we argue that an open question answering (qa) formulation (i.e., answer the question "what is this"?), as done, e.g., by @xcite , is an ill-defined task for two reasons. 1) the complete set of admissible answers is not provided (e.g., admissible answers for the dog in figure 1 include keeshond, dutch barge dog, and wolfspitz). for objects with only a few synonym labels, one can provide all answer options but this does not scale to hundreds or thousands of objects. constrained decoding to only the admissible labels is computationally expensive for large label sets @xcite . we extensively evaluate 12 publicly available lvlms on foci and find that many of them like the popular llava 1.5 struggle with fine-grained object classification. we observe that models with similar performance on established benchmarks can yield quite different and uncorrelated results on foci, highlighting that fine-grained object classification is indeed a distinct skill for lvlms, and that foci should thus complement existing image understanding and reasoning benchmarks. comparing the models further, we observe that the scale of their (pre-)training data seems to impact their performance on foci significantly more than for image understanding tasks. a comparison with the underlying clip models used as the lvlms' image encoders shows that the encoder's zero-shot accuracy provides an upper bound for the lvlm, with the lvlm performance lagging drastically behind. this suggests that the alignment between the image encoder and llm in lvlms seems to be insufficiently semantically fine-grained. we finally perform controlled experiments to isolate the modeling and training decisions that impact the models' performance in foci. as is the case with other benchmarks, both larger llms and stronger image encoders improve results. most importantly, incorporating captions into the training data that explicitly name the downstream objects helps with classification. similarly, including fine-grained classification objectives to the training mix can improve models' foci performance. ## multiple-choice image classification image classification is a fundamental problem in computer vision with a plethora of datasets available. in this work, we focus on fine-grained object classification where models have to differentiate between several objects belonging to a specific domain, e.g., animal species or car models. we leverage existing datasets as resources for annotated data and frame object classification as a multiple-choice task with well-defined answer candidates. why multiple-choice? the standard formulation of object classification tasks for lvlms is via question answering, with open-ended answer generation @xcite . this formulation, we argue, represents an ill-posed problem for two main reasons: (1) the expected level of granularity in the object taxonomy that is expected as the answer is not defined, and is difficult to define in general (e.g., for the image from figure 1 , dog, spitz, or keeshond are all correct labels); (2) the set of admissible answers in existing datasets is not complete: most objects have multiple synonymous labels, all of which constitute a correct answer (e.g., keeshond, dutch barge dog, and wolfspitz), but only subsets of those are provided as admissible labeles in existing datasets. providing complete synonym sets and specifying the expected level of granularity of the answer is, in the general case, infeasible, and while incorporating llms into the evaluation might alleviate some issues even with incomplete answers @xcite , this would greatly increase the cost of evaluating models. instead, we propose to formulate fine-grained object classification as a multi-choice task, where the models are provided with a set of candidate answers from which the correct answer is to be selected; this way the expected (i.e., correct) output is well-defined. ## evaluating public lvlms we evaluate 12 diverse and publicly available lvlms on foci. we then analyze how the performance of lvlms relates to the results of their underlying clip image encoders. ## controlled experiments we next perform a set of controlled experiments to disentangle the effects of individual lvlm design choices on (fine-grained) object classification. our analysis encompasses three main factors: (1) the llm size, (2) the image encoder, and (3) targeted changes to the training data. for ( 2 ) and ( 3 ), we train lvlms following the llava 1.5 recipe with stablelm 2 zephyr 1.6b @xcite as llm and openai clip-l/14-224 as the image encoder (see the appendix b for training details). ## conclusion in this work, we evaluate the capabilities of lvlms for fine-grained object classification over different domains. we address the ambiguity of openended qa-based object classification evaluation and propose to replace it with a multiple-choice formulation, in which we retain the task difficulty by mining difficult (semantically closest classes) choices with a clip model. this way, we create foci, a novel benchmark consisting of 9 finegrained multi-choice object classification datasets. we benchmark 12 public lvlms, demonstrating that their performance on foci is largely uncorrelated with that on other image understanding and reasoning benchmarks: this renders fine-grained object classification a skill that is complementary to what the existing benchmarks test the lvlms for. our ablations identify the quality of the image encoder and the amount of explicit caption mentions of image objects in lvlm training data as factors that drive the performance. we hope our work stimulates wider research efforts on improving lvlms for fine-grained object classification, in particular conceptual innovation (e.g., more effective training data and protocols for object classification with lvlms) that goes well beyond mere scaling of lvlm pretraining to billions of image-text pairs.