VIPHY : Probing “Visible” Physical Commonsense Knowledge (2023)

Abstract: Vision-language models (VLMs) have shown remarkable performance on visual reasoning tasks (e.g. attributes, location). While such tasks measure the requisite knowledge to ground and reason over a given visual instance, they do not, however, measure the ability of VLMs to retain and generalize such knowledge. In this work, we evaluate VLMs’ ability to acquire “visible” physical knowledge – the information that is easily accessible from images of static scenes, particularly along the dimensions of object color, size, and space. We build an automatic pipeline to derive a comprehensive knowledge resource for calibrating and probing these models. Our results indicate a severe gap between model and human performance across all three dimensions. Furthermore, we demonstrate that a caption pretrained LM significantly outperforms VLMs on both size and spatial tasks – highlighting that despite sufficient access to ground language with visual modality, they struggle to retain such knowledge.

## introduction the ability to reason and acquire knowledge from experience, while being intuitive for humans, has been a long-standing challenge for ai agents @xcite . examples such as the color of grass, or the relative position of monitor and table, are formally regarded as commonsense knowledge @xcite . the retention of such knowledge in humans is achievable due to the presence of long-term memory, broadly classified into episodic and semantic memory @xcite . while the former stores the information pertaining to personal events, the latter is geared towards general, decontextualized knowledge. 1 prior studies (greenberg and verfaellie, 1 for instance, the memory of one's birthday cake is episodic, whereas knowing that most birthdays include a cake is part of semantic memory. 2010) have acknowledged the interdependency between them, particularly the consolidation of semantic knowledge from episodic memories -aids humans to acquire commonsense from experience. pretrained language models @xcite have demonstrated the capacity to reason @xcite and retain knowledge @xcite . likewise, vision-language models @xcite driven by the availability of largescale paired image-text datasets have shown strong performance on visual reasoning tasks @xcite . while such tasks emphasize the model's ability to draw inferences from a specific visual instance -primarily to ground entities and reason about their attributes and relations, they do not, however, explicitly measure the consolidation of such knowledge. 2 in this work, we evaluate the model's ability to generalize aspects of grounding and reasoning tasks, regarded as commonsense knowledge. prior works have been largely directed towards probing language models pertaining to object properties such as weight, size, speed, and affor-dance @xcite . drawing upon the notion of world scopes @xcite , we find that such datasets, albeit comprehensive across aspects of physical knowledge, are ideally suited for embodied agents capable of interacting with the physical environment. this motivates us to develop resources that better align with the world scope of existing ai systems, primarily vision-language models. in this work, we introduce viphy, a visible physical commonsense dataset designed to probe aspects of physical knowledge that are easily accessible in images of static scenes. therefore, it can be argued that models pre-trained on such data have sufficient access to the "visible world". we build a large-scale dataset along three dimensions of objects: (1) color, (2) size, and (3) space. in contrast to prior works @xcite , we bypass crowdsourced annotations in favor of an automated pipeline to derive a resource spanning 14k objects (30×) from raw images. this is achieved by extracting object subtypes -informed by the visual context in images (e.g. kitchen sink). we leverage image data, along with existing visionlanguage and depth perception models to develop viphy. beyond scale, we introduce a resource for probing spatial knowledge of common environments. although one can reason along several types of spatial relations for a visual instance (e.g. a cat behind a laptop; liu et al. (2022a)), we find that mapping them to commonsense knowledge is nontrivial. 3 we define spatial relations by selecting "ground" as the observer and specifying the relative elevation of objects under an allocentric reference frame @xcite . we probe state-of-the-art models on viphy, and find a significant gap across all three dimensions, compared to human performance. previous works @xcite have corroborated the improvements from language grounding towards acquiring visual knowledgeour results, however, show a more nuanced picture. while vlms fare much better than lms on recalling colors, the caption pretrained baseline @xcite significantly outperforms vlms on both size and spatial inference tasks. this highlights that despite access to visual modality, existing vlms struggle to effectively consolidate such 3 due to challenges in specifying the reference frame of the observer, the canonical pose of the objects, and the situational nature of a scene. knowledge. the contributions of this work can be summarized as follows: (1) we build a comprehensive dataset, covering multiple aspects of visually accessible knowledge ( §3), which is developed through an automated pipeline to derive high-quality resources from images at scale ( §2). (2) we conduct extensive benchmarking across several state-of-theart language and vision-language models and find significant gaps in human performance ( §4). (3) we demonstrate a baseline tuned on the caption that significantly outperforms its vision-language counterparts -highlighting that despite access to images, vlms struggle to consolidate such knowledge. ## pipeline we provide a conceptual overview of our pipeline for developing viphy, as illustrated in fig. 3 . during the preprocessing stage, we build an internal database comprising object names and corresponding subtype candidates. given image and object regions as input foot_1 , we substitute object names with their subtypes ( §2.1), and compute the corresponding depth map. the processed data is used to extract color, size and spatial knowledge ( §2.2). ## dataset this section details the specific data sources and models used to develop viphy ( §3.1). we also report the dataset statistics and task format for each dimension ( §3.2). additional parameters related to dataset construction are provided in appendix a.1. ## experiments we evaluate several state-of-the-art models under zero-shot and finetune settings ( §4.1), and conduct further analysis of model performance ( §4.2). the datasets are partitioned into 20% train, 10% dev, 70% test set. baselines we consider the following language (lm) and vision-language (vlm) models: capbert in addition to the aforementioned baselines, we explore to what degree does an lm pretrained only on image captions, encodes visual knowledge. such a model effectively serves as a diagnostic baseline -ablating the grounding mechanism in vlms. we build capbert by pretraining bert base foot_3 on captions (4m) from coco @xcite , cc3m @xcite and vg @xcite datasets. while a contemporaneous work by @xcite developed a similar baseline, we find limited comparisons on color and no evaluation on their size dataset. through extensive benchmarking, we are able to derive novel insights with capbert ( §4.1). finetuning to evaluate under finetune setting, we train a linear classifier on top of the model's output, while rest of the weights are frozen. we use softmax cross entropy loss for single and multilabel setups, following @xcite . all probes are finetuned for 50 epochs, with batch size of 8, using adam optimizer @xcite and a learning rate of 10 -4 . prompts for probing lms and vlms, we provide manually designed textual prompt as input to the model. the prompt templates for probing across color, size and spatial tasks, under zero-shot (zs) and finetune (ft) settings are given in metrics we introduce the following metrics for measuring multi-label task performance: task setting prompt color zs o is of [mask] color ft [cls] color of o qa what is the color of o? (a) .. (b) .. size zs o1 is [mask] than o2 in size ft [cls] size of o1 in comparison to o2 qa what is the size of o1 in comparison to o2? (a) .. (b) .. spatial zs in a s, the o1 is located [mask] the o2 ft [cls] in a s, the o1 is located in comparison to o2 qa in a s, where is o1 is located in comparison to o2? (a) .. (b) .. here, d denotes samples in the evaluation set. in addition to the aforementioned metrics, we also report the macro-averaged f1-score (f1). human performance to provide an upper bound on viphy tasks, we use coda @xcite for color -computed over 432 overlapping objects. 10 for size and spatial tasks, we evaluate 100 relations with three external annotators 11 (crowdsourced) and report the average scores. ## conclusion we present viphy, a large scale resource for probing "visible" physical knowledge -information easily accessible from images of static scenes, across dimensions of color, size and space. we design an automated pipeline to extract and consolidate such knowledge facts from images, and introduce a new resource for evaluating spatial knowledge of common environments. our benchmarking evaluation highlights a huge gap between model and human performance across all three tasks. furthermore, while prior works have reported vlms to be more effective, our caption pretrained baseline (capbert) significantly outperforms vlms on the ability to recall size and spatial knowledge. these results underscore that despite access to visual modality, existing vlms struggle to retain visual knowledge as effectively.