An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models (2024)

Abstract: Large Multimodal Models (LMMs) have achieved strong performance across a range of vision and language tasks. However, their spatial reasoning capabilities are under-investigated. In this paper, we construct a novel VQA dataset, Spatial-MM, to comprehensively study LMMs’ spatial understanding and reasoning capabilities. Our analyses on object-relationship and multi-hop reasoning reveal several important findings. Firstly, bounding boxes and scene graphs, even synthetic ones, can significantly enhance LMMs’ spatial reasoning. Secondly, LMMs struggle more with questions posed from the human perspective than the camera perspective about the image. Thirdly, chain of thought (CoT) prompting does not improve model performance on complex multi-hop questions involving spatial relations. Moreover, spatial reasoning steps are much less accurate than non-spatial ones across MLLMs. Lastly, our perturbation analysis on GQA-spatial reveals that LMMs are much stronger at basic object detection than complex spatial reasoning. We believe our new benchmark dataset and in-depth analyses can spark further research on LMMs spatial reasoning.

## introduction large multimodal models (lmms) have shown impressive generalization ability on several vision and language tasks. several recent works, however, showed that these models lack spatial understanding @xcite @xcite @xcite . as can be seen in figure 1 , multimodal llms, including gpt-4o, often fail to answer questions from a human perspective within an image. the focus of this work is to study the understanding of spatial relations by top-performing lmms. moreover, we go beyond evaluating only the final answers to directly analyzing the intermediate reasoning steps generated by chain of thought (cot) prompting in multi-hop visual question-answering (vqa) tasks. we ground lmms ' reasoning steps into a scene graph format and verify whether they form a valid path. more specifically, we ask the following questions: (i) what spatial relations are missed by models, and why it happen? (ii) how can additional symbolic visual information, such as bounding boxes or scene graphs, improve the performance of lmms? which of these symbolic information is more useful, and how can they be integrated in the reasoning process effectively? (iii) how does the questions complexity affect lmms in handling spatial relations? (iv) how does the reasoning path of lmms behave when they fail to answer a multihop question? is the failure due to incorrect spatial reasoning or non-spatial reasoning? to address these questions, we construct spatial-mm, a novel, challenging dataset, and comprehensively lmms spatial reasoning capabilities from different angles. we analyze four top-performing lmms on spatial-mm and gqaspatial @xcite benchmarks to identify problems with visual question answering (vqa) evaluation methodology. our comprehensive analyses reveal a number of important insights that point to future research directions. our contributions can be summarized as follows. ## the spatial-mm benchmark we seek to study the spatial reasoning capability gap between humans and lmms. based on the observation that existing benchmarks only partially investigate the spatial reasoning capabilities of lmms @xcite , we introduce a novel benchmark, spatial-mm, which includes two subsets: spatial-obj and spatial-cot. spatial-obj features multiple-choice questions that focus on the spatial relationships between one or two objects in an image, while spatial-cot offers open-ended multi-hop questions. ## data enrichment to improve spatial reasoning intuitively, providing lmms with additional spatial-aware visual grounding data would help improve their spatial reasoning performance. to test this hypothesis, we describe two pipelines for generating different types of visual grounding data. the first pipeline consists of two stages: (i) key object extraction and (ii) bounding box generation. the second pipeline consists of three stages: (i) key object extraction, (ii) spatial-aware caption generation related to the key objects, and (iii) spatialaware scene graph generation. please refer to figure 6 in appendix c for detailed illustrations of our pipelines. furthermore, we describe reasoning question: where is the pink bear from the perspective of the woman wearing the floral dress? answer: right reasoning: [ {bear1 is pink}, {bear1 is on the left}, {woman1 wearing floral dress}, {women1's right is image's left}, {bear1 on the right from women1's perspective}] path generation for the multi-hop questions. ## conclusion motivated by our observation that lmms are not able to distinguish spatial relationships, in this paper, we propose a new benchmark to evaluate the spatial reasoning capabilities of lmms. our benchmark consists of two subsets: spatial-obj containing 2,000 multiple-choice questions to evaluate the spatial reasoning capabilities of one or two objects in a given image, spatial-cot containing 310 multihop questions to evaluate the spatial related cot and reasoning paths. we conduct comprehensive experiments on our benchmark and gqa-spatial. experimental results show the deficiencies in spatial reasoning of current lmms, including gpt-4o, gemini and llava-v1.5. we also find that bounding boxes and scene graphs are helpful in some cases to improve the lmms' prediction quality.