Visual Spatial Reasoning (2023)

Abstract: Spatial relations are a basic part of human cognition. However, they are expressed in natural language in a variety of ways, and previous work has suggested that current vision-and-language models (VLMs) struggle to capture relational information. In this paper, we present Visual Spatial Reasoning (VSR), a dataset containing more than 10k natural text-image pairs with 66 types of spatial relations in English (e.g., under, in front of, facing). While using a seemingly simple annotation format, we show how the dataset includes challenging linguistic phenomena, such as varying reference frames. We demonstrate a large gap between human and model performance: The human ceiling is above 95%, while state-of-the-art models only achieve around 70%. We observe that VLMs’ by-relation performances have little correlation with the number of training examples and the tested models are in general incapable of recognising relations concerning the orientations of objects.1

## introduction multimodal nlp research has developed rapidly in recent years, with substantial performance gains on tasks such as visual question answering (vqa) @xcite @xcite @xcite , vision-language reasoning or entailment @xcite @xcite , and referring expression comprehension @xcite . existing benchmarks, such as nlvr2 @xcite and vqa @xcite , define generic paradigms for testing vision-language models (vlms). however, as we further discuss in § 2, these benchmarks are not ideal for probing vlms as they typically conflate multiple sources of error and do not allow controlled analysis on specific linguistic or cognitive properties, making it difficult to categorize and fully understand the model failures. in particular, spatial reasoning has been found to be particularly challenging for current models, and much more challenging than capturing properties of individual entities @xcite @xcite , even for state-of-theart models such as clip @xcite . another line of work generates synthetic datasets in a controlled manner to target specific relations and properties when testing vlms, e.g., clevr @xcite and shapeworld @xcite . however, synthetic datasets may accidentally overlook challenges (such as orientations of objects which we will discuss in § 5), and using natural images allows us to explore a wider range of language use. to address the lack of probing evaluation benchmarks in this field, we present vsr (visual spatial reasoning), a controlled dataset that explicitly tests vlms for spatial reasoning. we choose spatial reasoning as the focus because it is one of the most fundamental capabilities for both humans and vlms. such relations are crucial to how humans organize their mental space and make sense of the physical world, and therefore fundamental for a grounded semantic model @xcite . the vsr dataset contains natural image-text pairs in english, with the data collection process explained in § 3. each example in the dataset consists of an image and a natural language description that states a spatial relation of two objects presented in the image (two examples are shown in figure 1 and figure 2 ). a vlm needs to classify the image-caption pair as either true or false, indicating whether the caption is correctly describing the spatial relation. the dataset covers 66 spatial relations and @xmath0, using 6,940 images from ms coco @xcite . situating one object in relation to another requires a frame of reference: a system of coordinates against which the objects can be placed. drawing on detailed studies of more than forty typologically diverse languages, @xcite concludes that the diversity can be reduced to three major types: intrinsic, relative, and absolute. an intrinsic frame is centered on an object, e.g., behind the chair, meaning at the side with the backrest. a relative frame is centered on a viewer, e.g., behind the chair, meaning further away from someone's perspective. an absolute frame uses fixed coordinates, e.g., north of the chair, using cardinal directions. in english, absolute frames are rarely used when describing relations on a small scale, and they do not appear in our dataset. however, intrinsic and relative frames are widely used, and present an important source of variation. we discuss the impact on data collection in § 3.2, and analyse the collected data in § 4. we test four popular vlms, i.e., visual-bert (li et al., 2019), lxmert @xcite , vilt @xcite , and clip @xcite on vsr, with results given in § 5. while the human ceiling is above 95%, all four models struggle to reach 70% accuracy. we conduct comprehensive analysis on the failures of the investigated vlms and highlight that (1) positional encodings are extremely important for the vsr task; (2) models' byrelation performance barely correlates with the number of training examples; (3) in fact, several spatial relations that concern orientation of objects are especially challenging for current vlms; and (4) vlms have extremely poor generalization on unseen concepts. ## dataset creation in this section we detail how vsr is constructed. the data collection process can generally be split into two phases: (1) contrastive caption generation ( § 3.1) and (2) second-round validation ( § 3.2). we then discuss annotator hiring and payment ( § 3.3), dataset splits ( § 3.4), and the human ceiling and agreement of vsr ( § 3.5). ## dataset analysis in this section we compute some basic statistics of our collected data ( § 4.1), analyze where human annotators have agreed/disagreed ( § 4.2), and present a case study on reference frames ( § 4.3). ## experiments in this section, we test vlms on vsr. we first introduce baselines and experimental configurations in § 5.1, then experimental results and analysis in § 5.2. then we discuss the role of frame of reference using experiments in § 5.3 and finally conduct sample efficiency analysis in § 5.4. ## conclusion and future directions we have presented visual spatial reasoning (vsr), a controlled probing dataset for testing the capability of vision-language models (vlms) of recognising and reasoning about spatial relations in natural image-text pairs. we made a series of linguistic observations on the variability of spatial language when collecting vsr. we highlighted the diverse use of reference frames among annotators, and also the ambiguous nature of certain spatial relations. we tested four popular vlms on vsr, and found they perform more than 25% below the human ceiling. on a more challenging concept zero-shot split, the tested vlms struggled to reach 60% accuracy and their performance plateaued even with increased training examples. among the finetuning-based vlms, vilt, and lxmert outperformed visualbert, and we noted out that the explicit positional information in the former two models is crucial in the task. clip with prompt engineering achieved slightly better than random performance, suggesting poor capability in spatial reasoning. we also performed a by-relation analysis and found that the models' performance on certain relations have little correlation with the number of training examples, and certain relations are inherently more challenging. we identified orientation as the most difficult category of relations for vlms. proximity is another challenging category, especially in the zero-shot setup as this relation is highly concept-dependent. we hope the task serves as a useful tool for testing and probing future vlms. in future work, we plan to more extensively investigate whether large-scale pretrained dualencoders such as clip @xcite , align @xcite , and lit ( zhai et al @xcite and test multilingual vlms. along the same line, since we have also collected the metadata of annotators, the vsr corpus can be used as a resource for investigating research questions such as: how is ''space'' described among different dialects of english? how is ''space'' perceived among different populations? we hope that the annotation process of vsr can also serve as a basis for future cross-lingual and cross-cultural sociolinguistic research.