Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models (2024)

Abstract: Despite recent advances demonstrating vision- language models’ (VLMs) abilities to describe complex relationships among objects in images using natural language, their capability to quantitatively reason about object sizes and distances remains underexplored. In this work, we introduce a manually annotated benchmark of 241 questions across five categories specifically designed for quantitative spatial reasoning, and systematically investigate the performance of SoTA VLMs on this task. Our analysis reveals that questions involving reasoning about distances between objects are particularly challenging for SoTA VLMs; however, some VLMs perform significantly better at this task than others, with an almost 40 points gap between the two best performing models. We also make the surprising observation that the success rate of the top-performing VLM increases by 19 points when a reasoning path using a reference object emerges naturally in the response. Inspired by this observation, we develop a zero-shot prompting technique, SpatialPrompt, that encourages VLMs to answer quantitative spatial questions using references objects as visual cues. Specifically, we demonstrate that instruct- ing VLMs to use reference objects in their reasoning paths significantly improves their quantitative spatial reasoning performance, bypassing the need for external data, architectural modifications, or fine-tuning. Remarkably, by solely using SpatialPrompt, Gemini 1.5 Pro, GPT-4V, and GPT-4o improve by 56.2, 28.5, and 6.7 points on average in Q-Spatial Bench without the need for more data, model architectural modifications, or fine-tuning.

## introduction spatial reasoning is essential for humans to interact with the world, such as determining if there is enough room on a desk for a backpack; if there is enough space to navigate through a room without hitting any obstacles; or if an object is placed sufficiently high enough to be inaccessible to a child. 1 project website: https://andrewliao11.github.io/ spatial_prompt spatial reasoning skills are also essential for visual ai agents in interactive applications such as visual question answering, augmented reality, robotics, or visual assistants @xcite . despite this need, the ability for an off-the-shelf vision language model (vlm) to perform finegrained spatial reasoning remains under-explored. current benchmarks primarily assess whether these models understand qualitative concepts like "left" versus "right" or "near" versus "far" from monocular images; formulating these tasks as questionanswering @xcite @xcite . however, recent studies have revealed that state-of-theart vlms struggle with quantitative spatial tasks. while measuring sizes or distances from monocular images is ill-posed @xcite , humans are surprisingly adept at making such estimations by relying on contextual clues @xcite , suggesting the potential improvements in vlms. to benchmark quantitative spatial reasoning, spatial vqa @xcite and @xcite are proposed. yet, both benchmarks either heavily relies on metric depth estimation or on the coarse label representations, e.g., 3d cuboids, potentially leading to inaccurate quantitative labels. on the other hand, to improve quantitative spatial reasoning in vlms, spatial vlm and spatialrgpt demonstrate that fine-tuning vlms with additional spatial vqa data and architectural modifications can enhance their performances significantly. however, this requires direct access to the model, as well as a significant amount of data and computational resources. in this paper, we introduce a new question answering corpora, q-spatial bench, specifically designed to evaluate quantitative spatial reasoning in vlms with high-precision. we identify the strengths and weakness across models by systematically analyzing the performance of sota vlms. in we demonstrate that by solely using spatialprompt, gemini 1.5 pro, gemini 1.5 flash, and gpt-4v improve their success rates by over 47, 22, and 30, respectively, in q-spatial bench without the need for more data, model architectural modifications, or fine-tuning. to put these numbers in perspective, we note that the pioneering work, spatialvlm @xcite , was fine-tuned on a massive dataset with 2 billion direct spatial reasoning qa pairs to achieve a relative improvement of less than 4 points in the same metric. while their evaluation benchmark is not publicly available for a direct comparison, we could still use this number as a proxy for reference. we emphasize our improvement is achieved purely with simple instructions using natural language. there is no external training data, model architectural modifications, or fine-tuning. most importantly, we believe our approach of analyzing emergent reasoning paths in high-performing models opens new horizons for transferring capabilities to less performant ones in a cost-effective manner. ## analysis we first test four commercial vlms including gemini 1.5 pro, gemini 1.5 flash @xcite , gpt-4v (openai, 2024), and gpt-4o, to identify which question categories are more challenging or easier for vlms in sec. 4.1. we then statistically analyze the potential factors leading to the strong gpt-4o performances on all tasks in sec. 4.2. ## spatialprompt: eliciting reasoning paths with reference objects inspired by the analysis in sec. 4.2, we propose a prompting technique, spatialprompt, for quantitative spatial reasoning that explicitly encourages vlms to identify reference objects within the images. orthogonal to prior work @xcite , our prompting techniques involve no additional data generation, fine-tuning or architectural modifications. ## limitations in this section, we discuss the limitations of our work from two aspects: the proposed benchmark and the proposed prompting technique. q-spatial bench is limited by its size due to the time-consuming questions-answering annotation, especially the time spent to measure highprecision distances. in cases where the end points of the distance is ambiguous, the annotators are required to clicking around the nearby point cloud to ensure the measurement quality. for example, when annotating the shortest distance between two objects, to find the end points of the shortest distance can be time-consuming. as compared to prior works @xcite , our benchmark is around four times smaller. spatialprompt is essentially trading tokens to performances since the appending text prompt encourages vlms to generate the reasoning paths in a detailed manner. this makes it less likely to be applied in the application requires real-time inference. additionally, the motivation of spatialprompt is that we want to encourage vlms to use the context information to improve quantitative spatial reasoning. however, when the image consists of a clean background, it is possible that there is little visual cues in the image; therefore, adding spatialprompt hurts or confuses vlms. finally, as shown in sec. 5.2.3, spatialprompt results in inconsistent improvements for open-source vlms. it is unclear that if the proposed spatialprompt only works when the model scale is large enough. to summarize, q-spatial bench is mainly limited by its sizes due to the costly labeling process. on the other hand, spatialprompt can be limited by inference speed, image context and maybe model scales. ## conclusion in conclusion, our work introduces q-spatial bench, a novel benchmark tailored for evaluating vlms on quantitative spatial reasoning and that incorporates 5 different question categories. through systematic analysis, we have identified the strengths and weaknesses of current sota vlms in this task. furthermore, we have developed a prompt-based technique that leverages naturally emerging reasoning paths to enhance vlm performance on quantitative spatial tasks. our findings demonstrate that instructing vlms to use reference objects in their reasoning pathways significantly boosts their performance without the need for additional data or fine-tuning. gpt-4v tends to perform a very rough estimate directly. on the other hand, when prompted with zero-shot cot and spatialprompt, gpt-4v leverages surrounding objects for quantitative spatial reasoning. gemini 1.5 pro responses with different text prompts. in fig. 12 , we compare gemini 1.5 pro responses with different text prompts. among all the evaluated prompts, spatialpropmt is the only one that successfully elicit quantitative spatial reasoning in gemini-1.5-pro. failure cases of gpt-4o. fig. 13 shows the common failure case of gpt-4o. we find that gpt-4o have a tendency to use floor tiles as references, leading to inaccurate estimation at times.