I mage N et VC : Zero- and Few-Shot Visual Commonsense Evaluation on 1000 I mage N et Categories (2023)

Abstract: Recently, Large Language Models (LLMs) have been serving as general-purpose interfaces, posing a significant demand for comprehensive visual knowledge. However, it remains unclear how well current LLMs and their visually augmented counterparts (VaLMs) can master visual commonsense knowledge. To investigate this, we propose ImageNetVC, a human-annotated dataset specifically designed for zero- and few-shot visual commonsense evaluation across 1,000 ImageNet categories. Utilizing ImageNetVC, we benchmark the fundamental visual commonsense knowledge of both unimodal LLMs and VaLMs. Furthermore, we analyze the factors affecting the visual commonsense knowledge of large-scale models, providing insights into the development of language models enriched with visual commonsense knowledge. Our code and dataset are available at https://github.com/hemingkx/ImageNetVC.

## introduction with the breakthrough progress of large language models (llms) in recent years @xcite , llms are gradually adopted as general-purpose api interfaces (e.g., chatgpt 1 ). in addition to language, these intelligent agents, are further required to understand vision knowledge @xcite , especially the visual perception, which is crucial for real-world interactions such as commonsense reasoning @xcite , recipe generation @xcite , and robotic navigation @xcite . however, current studies lack a systematic evaluation on how well these widely-used llms and their variants are capable of visual understanding. recent research proposes to evaluate the visual capability of models through visual commonsense evaluation @xcite let me imagine... oh! samoyeds are white. what color is the fur of a samoyed? yes, it is smiling. is the dog in the picture smiling or not? visual commonsense can be captured through a series of related images. @xcite . as shown in figure 1 , visual commonsense evaluation aims to evaluate the model's understanding of commonly shared human knowledge about generic visual concepts, including color @xcite @xcite , spatial relations @xcite , relative sizes @xcite , etc. despite their insightful investigations, these studies still have the following limitations from two sides: 1) data side: some research mines visual commonsense attributes based on frequency distributions in plain text corpora, which diverges from human visual perception and exhibits additional textual bias @xcite ; 2) model side: most existing evaluations only focus on a specific model group, lacking a comprehensive exploration of various model families @xcite @xcite . in this work, we propose that similar to human beings, models can also answer intricate visual commonsense questions with related images (illustrated in figure 1 ). to this end, we introduce imagenetvc, a unified zero-and few-shot visual commonsense benchmark incorporating multiple sources of images (e.g., imagenet @xcite , search images, and synthetic images). from the data side, imagenetvc comprises 4,076 highquality qa pairs, encompassing 1,000 imagenet categories across diverse domains such as color, shape, material, component, etc. moreover, as a human-annotated dataset, imagenetvc utilizes human visual perception to identify shared attributes across relevant images, avoiding textual bias and providing data that is more closely aligned with human knowledge. from the model side, besides unimodal llms, imagenetvc also enables the evaluation of various visually-augmented language models (valms) to investigate the effect of visual grounding, which compensates for the lack of images in previous benchmarks. with imagenetvc, we conduct extensive evaluations on the most widely-used llms and valms. we benchmark the visual commonsense capabilities of various llms such as opt, llama, and falcon and assess the effect of visual grounding in valms with multiple sources of relevant images. we further analyze the co-founding factors that may affect the visual commonsense capability of models, such as model scale, in-context learning, and image sources. we highlight several experimental findings. these findings support the high value of our benchmark in assessing visual commonsense capabilities. ## related work large language models text-only large language models (llms) have exhibited outstanding performance across various textual commonsense tasks, benefiting from their training on extensive textual data @xcite @xcite . however, the lack of vi-sual data (e.g., images) during pretraining restricts their visual commonsense capabilities @xcite . on the other hand, visually-augmented language models (valms) have gained popularity by integrating visual information into llms @xcite , which enhance the visual understanding capabilities of language models @xcite . visual commonsense evaluation visual commonsense knowledge of visual concepts is a fundamental and critical aspect of ai systems seeking to comprehend and reason about the world @xcite . previously, several datasets have been proposed to address specific attributes of visual commonsense, including memo-rycolors @xcite , colorterms @xcite , relativesize @xcite , and spatial commonsense (spatialcs) @xcite . to evaluate general visual commonsense, zhang et al. (2022a) introduced vicomte, a template-based dataset consisting of various (subject, object) pairs (such as (sky, blue)). however, its reliance on pure textual input underestimates the visual capabilities of valms. furthermore, its utilization of template-based formats and automatic extraction techniques leads to substandard data quality and inherent textual biases. in this work, we introduce imagenetvc, a human-annotated visual commonsense evaluation dataset that consists of 4k natural language qa pairs across various visual attributes, which supports both llm and valm evaluation with multiple sources of images. we present detailed comparisons of imagenetvc with prior work in ## imagenetvc starting from imagenet, we construct our ima-genetvc dataset in a multi-step crowd-sourcing pipeline, including 1) annotator training, 2) commonsense annotation, and 3) cross-check examination. an overall demonstration of our annotation process is illustrated in figure 2 . ## experiments our experiments primarily focus on two types of language models: llms and valms. both models have demonstrated promising capabilities in understanding visual information @xcite . large language models we begin with the textonly setting, to explore the visual commonsense capability of llms learned from the textual corpus. we focus on the dominant auto-regressive llm family and benchmark a lot of model variants, including the gpt @xcite @xcite , opt @xcite , llama @xcite , falcon @xcite , and pythia @xcite . ## analysis we further investigate the factors influencing the visual commonsense capabilities of llms and valms. for instance, we find that a decent scale (e.g., 1.3b) could be a potential threshold for textonly llms to learn visual commonsense. we then analyze several influencing factors of valms, such as image sources and image numbers. ## visual commonsense in other models it is worth noting that, as a general visual commonsense dataset, imagenetvc supports various types of models and evaluation settings. except for the evaluation setting in our main results, we also evaluate several models in the setting of open-ended generation. specifically, we select two widely-used multimodal models, 12-in-1 @xcite and blip @xcite that are finetuned on the vqav2 dataset @xcite and the famous rlhf model, chatgpt (gpt-3.5-turbo) for evaluation. 9 as illustrated in ## conclusion in this paper, we introduced imagenetvc, a comprehensive human-annotated dataset for evaluating visual commonsense using both textual and visual inputs. we conducted extensive experiments to evaluate the visual commonsense of both unimodal llms and valms using imagenetvc. our results demonstrate the varying degrees of visual commonsense knowledge present in different models, as well as the factors that contribute to the acquisition and enhancement of this knowledge. additionally, we offer insights into the emergent abilities of llms and the strengths of valms in the realm of visual commonsense.