T op V iew RS : Vision-Language Models as Top-View Spatial Reasoners (2024)

Abstract: Top-view perspective denotes a typical way in which humans read and reason over different types of maps, and it is vital for localization and navigation of humans as well as of ‘non-human’ agents, such as the ones backed by large Vision-Language Models (VLMs). Nonetheless, spatial reasoning capabilities of modern VLMs in this setup remain unattested and underexplored. In this work, we study their capability to understand and reason over spatial relations from the top view. The focus on top view also enables controlled evaluations at different granularity of spatial reasoning; we clearly disentangle different abilities (e.g., recognizing particular objects versus understanding their relative positions). We introduce the TopViewRS (Top-View Reasoning in Space) dataset, consisting of 11,384 multiple-choice questions with either realistic or semantic top-view map as visual input. We then use it to study and evaluate VLMs across 4 perception and reasoning tasks with different levels of complexity. Evaluation of 10 representative open- and closed-source VLMs reveals the gap of more than 50% compared to average human performance, and it is even lower than the random baseline in some cases. Although additional experiments show that Chain-of-Thought reasoning can boost model capabilities by 5.82% on average, the overall performance of VLMs remains limited. Our findings underscore the critical need for enhanced model capability in top-view spatial reasoning and set a foundation for further research towards human-level proficiency of VLMs in real-world multimodal tasks.

## introduction large language models (llms) such as llama 2 and 3 @xcite , mistral @xcite , and gpt models (openai, 2022) have delivered impressive performance across a range of text-based tasks and applications such as question * equal contribution answering, language generation, and arithmetic reasoning @xcite . building on these text-only llms, the so-called vision language models (vlms), equipped with the capability to process and reason over multimodal vision-language information, have enabled multi-modal processing @xcite . they ground language reasoning ability of llms into the information of different modalities @xcite . prominent examples of vlms such as llava @xcite , @xcite , and gemini (google, 2024), have demonstrated strong performance across applications such as visual question answering @xcite , image captioning @xcite , and object grounding @xcite . spatial reasoning, one of the fundamental desirable properties of and requirements for vlms, has also gained increased attention recently @xcite @xcite . it requires grounding the model's reasoning ability with natural language into its visual perception of the surrounding environment @xcite . in particular, it involves two critical steps: (i) interpreting the environment visually, and (ii) reasoning over spatial relations. as a fundamental ability for the model to recognize, understand, and navigate through the physical world, it plays a crucial role in various downstream tasks such as vision-language generation @xcite and embodied ai @xcite . however, previous research has focused on exploring spatial reasoning abilities of vlms only from a conventional first-person perspective view @xcite . in this work, we aim to study and evaluate spatial understanding and reasoning capability of vlms from the top-view perspective, also referred to as the bird's-eye view @xcite . when compared to the conventional perspective view, top view offers better natural alignment: it is the typical perspective used for reading maps or presenting floor plans. moreover, it is inherently more complex: top-view maps encapsulate a wealth of information about different scenes, locations, objects and their relationships in the environment based on a single image. in addition to the photorealistic top-view maps, semantic top-view maps @xcite use different colors to represent different types of objects; we run experiments with both map types, see figure 1 . one advantage of top-view maps is that they define a controlled and interpretable experimental framework. indoor scenes, which are the focus of this work, typically feature a relatively stable set of objects and layouts, making them ideal for controlled studies. this allows us to disentangle and investigate various aspects of spatial reasoning and vlms' capabilities in a controlled manner. 1 ## task definition following prior work @xcite , we frame all tasks as multiple-choice qa tasks. given a top-view (realistic or semantic) map of a room m , the model must choose the correct option o i from the four options provided @xmath0, o 1 , o 2 , o 3 } that answers the question. 4 this format simplifies the evaluation and interpretation of the results. top-view maps. we provide two different types of top-view maps to the models: realistic maps m real and semantic maps m sem . realistic maps are constructed by placing a simulated orthographic camera above the scene to capture a photo-realistic top-view image. semantic maps represent objects in the scene with colored bounding boxes. each object is assigned a specific color and labeled at the same relative coordinates on the map to preserve the object's semantic information and spatial allocation. in comparison to realistic maps, semantic maps simplify the initial step of spatial reasoning (i.e., environment interpretation) by labeling the object types with corresponding colors and excluding irrelevant additional details such as shape and texture found in realistic top-view maps. given the customizable and flexible nature of colorobject mapping, the semantic map can also serve as an ideal testbed for evaluating models' out-ofdistribution (ood) performance, thereby encouraging further exploration beyond the scope of this work. example maps are in figure 1 . tasks and sub-tasks. we define 4 different tasks which cover a total of 9 finer-grained sub-tasks, with concrete examples shown in figure 1 . the tasks are designed to have an increasing level of complexity, where each subsequent task depends on the abilities measured in the preceding one(s). ## topviewrs dataset in order to study and evaluate the abilities of stateof-the-art vlms on the 4 tasks spanning 9 subtasks from §3, we now introduce a novel evaluation dataset, topviewrs, which focuses on top-view maps of indoor scenes (i.e., houses and rooms), discussed in what follows. dataset features. it introduces several advancements and innovative features that distinguish it from all prior visual spatial reasoning datasets. 1) multi-scale top-view maps: the selected topview maps of indoor scenes (see figure 1 ) provide a more natural representation of spatial environments that aligns with human cognitive map @xcite . this makes benchmarking spatial awareness more straightforward and meanwhile mitigates spurious correlations in the positions between objects commonly found in realistic front-view images. compared to the front view, the multi-scale top-view maps of single rooms and full houses add more divergence in the granularity of the entities (objects or rooms) in spatial reasoning. meanwhile, we provide both realistic maps and semantic maps for more comprehensive evaluation. ## conclusion in this study, we designed four tasks to examine the capabilities of vlms as top-view spatial reasoners, progressing from basic top-view map comprehension to dynamic spatial reasoning along navigation paths. to enable investigation into topview spatial reasoning abilities of vlms, we collected a novel dataset, topviewrs, which includes 11,384 multiple-choice questions, featuring photo-realistic and semantic top-view maps as the visual input. our extensive experiments involved evaluating 10 vlms across various model families and sizes on topviewrs. the results highlight a critical observation: particularly in complex reasoning tasks, vlms frequently perform only as well as a random baseline, with even more pronounced deficits when handling tasks with semantic maps. moreover, there is a noticeable performance gap compared to human annotators, underscoring the significant potential for further improvements in this area. in response to these findings, we discovered that employing chain-of-thought reasoning enhances model performance in spatial reasoning by 5.82%. despite this progress, the overall performance of vlms on spatial reasoning remains less than satisfactory. we hope that our study can set the stage for future research in multimodal spatial reasoning and encourage further investigations into refining the reasoning techniques, moving vlms closer to human-level proficiency in understanding and reasoning over real-world environments.