What’s “up” with vision-language models? Investigating their struggle with spatial reasoning (2023)

Abstract: Recent vision-language (VL) models are powerful, but can they reliably distinguish “right” from “left”? We curate three new corpora to quantify model comprehension of such basic spatial relations. These tests isolate spatial reasoning more precisely than existing datasets like VQAv2, e.g., our What’sUp benchmark contains sets of photographs varying only the spatial relations of objects, keeping their identity fixed (see Figure 1: models must comprehend not only the usual case of a dog under a table, but also, the same dog on top of the same table). We evaluate 18 VL models, finding that all perform poorly, e.g., BLIP finetuned on VQAv2, which nears human parity on VQAv2, achieves 56% accuracy on our benchmarks vs. humans at 99%. We conclude by studying causes of this surprising behavior, finding: 1) that popular vision-language pretraining corpora like LAION-2B contain little reliable data for learning spatial relationships; and 2) that basic modeling interventions like up-weighting preposition-containing instances or fine-tuning on our corpora are not sufficient to address the challenges our benchmarks pose. We are hopeful that these corpora will facilitate further research, and we release our data and code at https://github.com/amitakamath/whatsup_vlms.

## introduction pre-trained vision-language models perform well on complex tasks such as vqav2 @xcite and nocaps @xcite , even in the zero-shot setting @xcite . however, recent work has re-surfaced a concern that has long plagued vision-language models @xcite : new multimodal models still exhibit poor behavior on simple tasks like attribute attachment, counting, etc. @xcite @xcite . despite improvements, models still fail to reliably capture even a dog on a table ! " a dog right of a table a dog left of a table a dog right of a table a dog right of a table a dog right of a table a dog under a table a dog right of a table figure 1 : we propose three tightly controlled benchmarks to assess model capacity for fine-grained spatial reasoning, showing that popular vision-language models fall far behind human performance when asked to select the correct spatial relation between two objects in an image (real examples shown). ## benchmarks existing benchmarks include spatial reasoning questions, such as vqav2 @xcite and @xcite . however, instances in these corpora often conflate several types of reasoning: in gqa, over 92% of the validation questions do so. for example, the gqa question "are there men to the left of the person that is holding the umbrella?" conflates evaluation of spatial reasoning, object relationships, and object detection -in contrast, our questions require only spatial reasoning about one or two objects. our three new evaluation corpora are presented in the same format: an image paired with several captions which differ only by a preposition. what'sup consists of tightly controlled photographs we captured ourselves, whereas cocospatial and gqa-spatial are curated from wellrecognized image datasets. one key contribution is that all instances in all of our corpora require only spatial reasoning about one or two objects, e.g., in what'sup, we circumvent the part-andwhole problem discussed in @xcite by careful construction. figure 2 contains examples of images from each of our three benchmarks, along with the caption options each image is paired with. ## data-informed attempts at improvement in this section, we operationalize our hypotheses detailed above to yield potential solutions to models' struggle with learning spatial relations. ## related work spatial reasoning has long been evaluated by visionlanguage benchmarks: vqav2 @xcite , gqa (hudson and manning, 2019), nlvr2 @xcite , clevr @xcite and shapeworld @xcite all contain questions requiring spatial reasoning. however, many of these questions conflate several types of reasoning. performance on these benchmarks therefore masks vl models' struggle with spatial understanding specifically. more recently, vision-language benchmarks evaluating more specific phenomena have been proposed, testing understanding of word order @xcite , counting @xcite , object-attribute association @xcite , and compositionality @xcite . other work including valse @xcite , vsr @xcite and reclip @xcite evaluate spatial reasoning in isolation, as we do in our three corpora, testing vl models' ability to match an image to the more fitting of two captions where only the spatial preposition is flipped. they show that models have room for improvement in both zero-shot and finetuned settings. however, all of the non-synthetic benchmarks above testing spatial reasoning are based on coco @xcite or visual genome @xcite , which are sourced from flickr. these images tend to have many objects, usually in cluttered environments, which can confuse models trained with only image-level supervision @xcite . the images also reflect biases in our usual world, such as mugs usually being on tables and not under them foot_3 . models may learn these priors and attain high scores on these benchmarks without actually attending to the images @xcite )e.g., text-only gpt-1 @xcite scores 27 accuracy points above random chance on spatial reasoning questions in valse. in contrast, we capture sets of photographs for what'sup which are uncluttered, unambiguous, and contain all four preposition options for any pair of objects -thus exposing any bias models may have for the "usual" relation between two objects, as well as preventing models with such a bias from leveraging it to mask their spatial understanding abilities. text-to-image generation has also been shown to struggle with correctly depicting spatial relations @xcite . our work sheds light on why this could be the case: e.g., dall-e 2 @xcite uses a frozen clip backbone, and as we show in our work, clip itself struggles with spatial reasoning. ## conclusion in this work, we propose three new benchmarks: what'sup, coco-spatial and gqa-spatial, to evaluate vl models on basic spatial relations in a range of environments, with the controlled nature of what'sup allowing us to evaluate pairs and sets of prepositions for a given object pair. we observe that all 18 models we evaluate perform poorly on these benchmarks in a zero-shot fashion. next, we study the laion dataset which was used to train openclip, revealing that prepositions are rare, ambiguous, and extraneous in the captions. finally, we explore potential remedies, ultimately finding that clip models, at least in the regime of scale we consider, fail to even fit a large-scale training set that requires precise spatial reasoning. how might models solve our newly proposed evaluations going forward? three promising future directions include: (1) auto-generation of hard negatives for spatial prepositions (and beyond) during pre-training; (2) consideration of more expressive fine-tuned models that support image-text crossattention and mixes of contrastive and generation objectives; and (3) thorough scaling experiments to probe for potentially promising relationships between increasing compute of vision-language models vs. performance on our benchmarks.