S pa RC and S pa RP : Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Models (2024)

Abstract: Spatial reasoning is a crucial component of both biological and artificial intelligence. In this work, we present a comprehensive study of the capability of current state-of-the-art large language models (LLMs) on spatial reasoning. To support our study, we created and contribute a novel Spatial Reasoning Characterization (SpaRC) framework and Spatial Reasoning Paths (SpaRP) datasets, to enable an in-depth understanding of the spatial relations and compositions as well as the usefulness of spatial reasoning chains. We found that all the state-of-the-art LLMs do not perform well on the datasets—their performances are consistently low across different setups. The spatial reasoning capability improves substantially as model sizes scale up. Finetuning both large language models (e.g., Llama-2-70B) and smaller ones (e.g., Llama-2-13B) can significantly improve their F1-scores by 7–32 absolute points. We also found that the top proprietary LLMs still significantly outperform their open-source counterparts in topological spatial understanding and reasoning.

## introduction spatial understanding and reasoning are a crucial component of both biological and artificial intelligence, essential for daily interactions and common tasks such as dialogues and conversations @xcite , navigation @xcite @xcite , and robotics @xcite , among others. they require common reasoning steps such as identifying objects, determining other objects being involved, and aggregating multiple spatial 1 code: https://github.com/ukplab/acl2024-sparc-andsparp dataset: https://huggingface.co/datasets/ukplab/sparp tu-datalib dataset: https://tudatalib.ulb.tudarmstadt.de/handle/tudatalib/4235 relations to reach a conclusion. the advancement of the field has significantly benefited from many well-known tasks and datasets, including babi @xcite , spartqa @xcite , spartun and resq @xcite , and stepgame @xcite , among others. recently, large language models (llms) have been shown to be capable of performing abstract, commonsense-based, and multi-hop reasoning @xcite @xcite . if such models are to be used as intelligent agents to answer questions, perform tasks, and collaborate with humans, whether they can understand the basic spatial relationships and perform corresponding reasoning would become critical to many real-life applications. in this work, we present an extensive study on the state-of-the-art llms' capability in spatial reasoning. the key components of spatial abilities include: (i) understanding spatial relations and composition, and (ii) developing reasoning chains to reach conclusions. prior work @xcite @xcite has focused on the relations and spatial composition tied to a limited context setup, as will be detailed later in this paper. in our work, we propose a bottom-up approach that builds upon detailed spatial properties, providing fine control for constructing spatial rules and context setups. we formalize and propose spatial reasoning characterization (sparc), a systematic framework in defining spatial properties of objects, relations, and contexts, as well as how they characterize spatial composition, which is inspired by the widely used benchmarks spartun @xcite and stepgame @xcite . reasoning paths are an integral part of the reasoning process and critical for analyzing and enhancing reasoning models. to the best of our knowledge, unlike other reasoning tasks such as mathematical reasoning, there exists no dataset with textual spatial reasoning paths. in this paper we develop deductively verified spatial reasoning paths by using spatial reasoners to generate stepby-step reasoning on spartun and stepgame, which is then verbalized to form textual chain-ofthoughts. we show that finetuning different sizes of llms (13b and 70b) on the reasoning paths significantly improves their spatial reasoning performance, which also highlights the poor performance of the generalist pretrained llms (without finetuning) on spatial reasoning. in summary, our contributions are as follows: ## the spatial reasoning characterization (sparc) framework the steps to identify and compose spatial relations between entities distinguish spatial reasoning from other reasoning tasks. prior work e.g. spartun @xcite and stepgame @xcite , have focused directly on the spatial composition rules coupled with the contexts, which can lead to different conclusions even for the same set of relations. for example, for the same context "a is left of b and b is above c", applying the spatial composition of stepgame concludes that a is to the left and above c, while no directional relation between a and c can be concluded at all by applying the spatial rules of spar-tun. the conclusions are completely different but equally valid. this difference can be reconciled by examining the underlying spatial properties of the objects and relations, specifically the treatment of objects as points vs extended, and completeness of the knowledge of relations in the context. we, therefore, advocate for an extendable bottom-up approach starting from a more granular level and introduce the spatial reasoning characterization (sparc) framework. sparc prioritizes spatial properties over spatial composition rules. consequently, it offers finer control in creating contexts and facilitates a deeper and systematic examination of the spatial reasoning capabilities. to keep our work closer and comparable to the widely used existing benchmarks, spar-tun @xcite and stepgame @xcite , we identify six properties that cover and characterize these datasets by two distinct and mutually exclusive sets of three properties each. with sparc, we further explore two properties sets (ps) with properties in common to these existing benchmarks. f sub-type relations (r) textual label (l) topological t r (rcc8) dc outside ec outside and touching po partially overlapping eq overlapping tpp inside and touching ntpp inside tppi contains and touches ntppi contains directional d r (relative) left left right right above above below below front front behind behind d c (cardinal) north above south below east right west left d t (clock) 12 o'clock above 3 o'clock right 6 o'clock below 9 o'clock left distance s q (qualitative) near near far far s u (quantitative) -- ## the spatial reasoning paths (sparp) reasoning paths are an integral part of reasoning models and critical for analyzing and enhancing such models. to the best of our knowledge, unlike other reasoning tasks such as mathematical reasoning, there exists no dataset with spatial reasoning paths. in this section, we develop deductively verified spatial reasoning paths by verbalizing the symbolic steps. existing spatial reasoning datasets can be considered as a collection of context-question-answer (c, q, a) tuples. formally, we denote a context @xmath0, r, t) i } n @xmath1, r, t) tuples, where @xmath2, @xmath3. for a given (c, q, a) tuple, seeking relation between the head h q and tail t q entities, we define a symbolic algorithm 1 relative direction composition for set of properties ps2 and ps3 in 2d. input: pairs to compose {pair1, pair2}. algorithm 2 relative direction composition for set of properties ps4 in 2d. input: pairs to compose {pair1, pair2}. current set of constraint inequalities ineq output: merged pair and updated inequalities ineq. we denote the extended dataset as spatial reasoning paths (sparp). specifically, we extended spartun, stepgame, sparc-ps3, and sparc-ps4, to be sparp-ps1, sparp-ps2, sparp-ps3 and sparp-ps4, respectively, by enriching the former with the reasoning paths. a comparison of the derived datasets with the original datasets is summarized in implementation details. to help replicability, we include implementation details such as dataset sampling, system prompt, and training parameters in appendix-c. ## results and analysis we run experiments with three state-of-the-art llms -llama-2-13b, llama-2-70b @xcite , and gpt-4 foot_1 , each one with both single greedy decoding and self-consistency @xcite with majority voting over 20 generations with sampling (@xmath5. inputs are provided with a "system prompt" containing task instructions and 5-shot cot with randomly sampled exemplars from the relevant dev-set, e.g., exemplars for a test instance of sparp-s-ps1 (spartun) were randomly sampled from its own dev-set. we also finetune llama-2 13b and 70b models, indicated by ft in overall results. as shown in we also observed that the spatial reasoning ability of llms improves significantly with increasing model sizes. the smaller pre-trained llama-2 13b model essentially exhibits no spatial reasoning ability, with the f1-scores of 0.49, 0.47, 0.92, and 1.84 on sparp-s-ps1 (spartun), sparp-s-ps2 (stepgame), sparp-s-ps3, and sparp-s-ps4, respectively. in contrast, the larger pre-trained llama-2 70b model demonstrates comparatively significant spatial reasoning ability, achieving f1-scores of 23.37, 26.41, 25.27, and 22.13 on sparp-s-ps1 (spartun), sparp-s-ps2 (stepgame), sparp-s-ps3, and sparp-s-ps4, respectively. ## conclusion spatial reasoning is one of the basic components of intelligence. we perform a study on the spatial reasoning abilities of the latest llms under comprehensive setups. to support the study, we introduce (sparc), a systematic framework to characterize spatial reasoning scenarios by identifying and defining six spatial properties of objects, spatial relations, and contexts, and their impact on the spatial composition rules. based on that, we create the (sparp) reasoning paths for the datasets. we found that the state-of-the-art llms do not perform well on the datasets -their performances are consistently low across different setups. the spatial reasoning capability improves significantly as model sizes scale up. finetuning both large language models (e.g., llama-2-70b) and smaller ones (e.g., llama-2-13b) can significantly improve their performance by 7-32 points on f1-scores. we also found top proprietary llms still significantly outperform their open-source counterparts in topological spatial understanding and reasoning. we provide detailed analyses and insights in our experiments.