E mb S patial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models (2024)

Abstract: The recent rapid development of Large Vision-Language Models (LVLMs) has indicated their potential for embodied tasks. However, the critical skill of spatial understanding in embodied environments has not been thoroughly evaluated, leaving the gap between current LVLMs and qualified embodied intelligence unknown. Therefore, we construct EmbSpatial-Bench, a benchmark for evaluating embodied spatial understanding of LVLMs. The benchmark is automatically derived from embodied scenes and covers 6 spatial relationships from an egocentric perspective. Experiments expose the insufficient capacity of current LVLMs (even GPT-4V). We further present EmbSpatial-SFT, an instruction-tuning dataset designed to improve LVLMs’ embodied spatial understanding.

## introduction embodied ai is the frontier direction of generalpurpose ai systems, requiring intelligent agents to understand instructions, perceive physical environments, plan and execute actions to accomplish corresponding tasks @xcite . recently, llm-based large vision-language models (lvlms) have demonstrated powerful capabilities in following instructions and performing planning based on the visual contexts @xcite @xcite , paving a promising path for the development of embodied ai systems. however, recent studies have revealed significant deficiencies of lvlms in understanding visual contents @xcite . in terms of embodied scenarios, the ability to understand spatial relationships between objects is particularly vital for agents to effectively interact with the environment @xcite . evaluating and enhancing such capabilities of lvlms is essential for constructing lvlm-driven embodied * equal contribution † corresponding author ## embspatial-sft to further improve lvlms' capacity in embodied spatial understanding, we construct an instructiontuning dataset, embspatial-sft, which provides qa data for two tasks: spatial relationship identification and object localization. the former task setting is consistent with embspatial-bench, while the latter serves as an auxiliary task to enhance the model's ability to ground target objects. the auxiliary task can be considered as the foundational skill for relationship identification. embspatial-sft is solely built on the training split of mp3d. in this way, we can still conduct zero-shot evaluations of the instruction-tuned models using data from the other two scenes in embspatial-bench. ## conclusion in this work, we propose embspatial-bench, a benchmark to evaluate embodied spatial understanding of lvlms. the evaluation results reveal the weak spatial understanding ability of current popular lvlms. we further propose embspatial-sft, an instruction tuning dataset to enhance the capacity of lvlms. extensive experiments valid the effectiveness of each data component in our embspatial-sft, with the goal of empowering the spatial understanding ability of lvlms.