A Benchmark for Reasoning with Spatial Prepositions (2023)

Abstract: Spatial reasoning is a fundamental building block of human cognition, used in representing, grounding, and reasoning about physical and abstract concepts. We propose a novel benchmark focused on assessing inferential properties of statements with spatial prepositions. The benchmark includes original datasets in English and Romanian and aims to probe the limits of reasoning about spatial relations in large language models. We use prompt engineering to study the performance of two families of large language models, PaLM and GPT-3, on our benchmark. Our results show considerable variability in the performance of smaller and larger models, as well as across prompts and languages. However, none of the models reaches human performance.

## introduction large language models (llms) are becoming increasingly human-like in their performance on many tasks, but are still not on par with more advanced aspects of human cognition @xcite . on the other hand, they are showing emerging capabilities that were previously thought beyond their limits, such as grounding conceptual spaces @xcite . currently, many questions are open regarding the limits of reasoning in llms and how they compare to humans in cognitive domains that require a deeper understanding of the world. one such domain is spatial reasoning, which is a fundamental part of human cognition @xcite @xcite . this type of reasoning is relevant not only for the representation, prediction and manipulation of physical objects, but also for representing and performing inferences with abstract concepts. this is reflected in common uses of spatial prepositions, which traditionally indicate location, but are also used to refer to abstract states, forces or goals. for example, one can be "in paris" or "under a tree" (physical locations), but one can also be "in trouble" or "under sedation" (abstract concepts). given their lack of embodied spatial experience and the scarcity of commonsense knowledge in training data @xcite , we hypothesise that llms have may difficulties reasoning about physical and abstract spatial relations. we investigate this using a novel benchmark for assessing inferences on sentences containing spatial prepositions. the sentences are designed to be easy for humans, but non-trivial for models that cannot differentiate between uses of prepositions with different concepts. our task has similarities with other nli tasks @xcite . this paper makes the following contributions: ## related work to investigate commonsense spatial reasoning, @xcite introduced a benchmark focused on assessing the relative size of objects, as well as positional relationships between humans and objects during various actions. commonsense spatial relationships. in contrast to these studies, our benchmark proposes the additional challenge of using spatial prepositions to refer to abstract concepts in addition to physical relationships. reasoning with metaphorical and literal statements has been previously studied @xcite , but here we focus specifically on spatial prepositions. ## dataset we create small, manually-curated datasets, intended to be used as a benchmark, and not for training purposes. each dataset consists of 400 class-balanced items. as illustrated in in the case of congruent compositions, the conclusion holds, typically indicating a similar type of spatial relationship. for example, if "john is in the crib" and "the crib is in the living room", the conclusion "john is in the living room" holds. on the other hand, in the incongruent compositions, the spatial prepositions in each premise refers to a different type of relation, such as through a conceptual metaphor, and the conclusion does not hold. however, the items are designed such that without a deep understanding of the commonsense semantics of the spatial prepositions, a mistaken interpretation is possible, leading to the false impression that the conclusion holds. for example, if "john is in the newspaper" and "the newspaper is in the kitchen", the conclusion "john is in the kitchen" does not hold. in this example, the spatial preposition "in" is used differently in the two premises: in the first premise, it refers to an abstract concept (inclusion as content in a newspaper), while in the second premise it refers to a physical location. hence, in this example, combining the premises does not validate the conclusion. the items are class-balanced: for every congruent item that uses prepositions {prep 1 , prep 2 , prep 3 } there is an incongruent item containing the same prepositions in sequence. we release datasets in english and in romanian. for both languages, each item was created by a native or a proficient speaker of the language, and always independently verified by another native speaker. in the process of creating items, we aimed to cover common cases for each chosen spatial preposition in order to create a representative sample of spatial preposition semantics. the creation of items was assisted by standard dictionaries with usage examples for each preposition. for a discussion on the limitations of the data generation process, please refer to section 7. in english, we use the spatial prepositions "in", "at", "on", "with", "under" , "above" and "behind". in romanian, we use their equivalents "în", "la", "pe", "cu", and "sub", respectively foot_1 . the use of prepositions is different in the two languages and hence the datasets are not direct translations of each other, but reflect the semantics of each language. the distribution of prepositions is shown in to validate the benchmark, we asked englishspeaking and romanian-speaking adults to answer dataset questions of the form "if {premise1} and {premise2}, does that imply that {conclusion}?" with "yes" or "no". the respondents were told that the aim was to collect a set of commonsense responses from humans and compare them to llms responses, which they consented to. the respondents were not paid. each respondent answered 20 randomly-selected questions from the dataset. as a response quality measure, we only included the responses where the accuracy for congruent questions, which we consider easier, was above 75%. we thus obtained responses from 27 englishspeaking and 23 romanian-speaking adults. this allows the detection of an effect size of 0.56 and 0.61, respectively, at alpha 0.05 and power 0.8. the results are shown in ## large language models evaluation we evaluated the performance of palm @xcite in different sizes: 540b, 62b (the original model, as well as the model trained to 1.3t tokens as explained in their appendix f) and 8b, as well as flan-palm-540b @xcite . we also evaluated gpt-3 @xcite ada (text-ada-001), babbage (text-babbage-001), curie (text-curie-001) and davinci (text-davinci-003). we prompted the models with questions of the form "if {premise1} and {premise2}, does that imply that {conclusion}?". we tested the llms with 0-shot, 1-shot and 5-shot prompts. in few-shot settings, each example was prefixed with 1 or 5 different randomly-selected examples from the dataset, each followed by its correct answer ("yes" or "no"). we assessed llms in a binary-choice setup of the benchmark. the models were asked to score the strings "yes" and "no" (and their romanian equivalents) given as candidate continuations to the above prompt. an example was labelled as correct if the log likelihood score of the correct continuation string was higher than the log likelihood score of the incorrect continuation. to mitigate prompt sensitivity @xcite , we used multiple prompt variations, as detailed in appendix a. we report the best prompt performance for each model and setup. for each best prompt, we obtained confidence intervals by randomly sampling sets of 20 responses, similarly to the format of the humans responses. as a baseline, we ran the same experiment using only the conclusion as a prompt, in the form: "{conclusion}?". this can probe whether the performance might be explained by the likelihood of the conclusion only. we report the results for the highest-scoring baseline value across all models. as an alternative to the binary-choice setup, our benchmark can also be used in a generative setting. this can be useful for assessing llms for openended or conversational applications. to illustrate this use of the benchmark, we performed a generative assessment of the largest model, palm-540b. the setup was identical to the above, except that the model was asked to generate 10 tokens in response to the given prompt, and responses were scored accordingly (see appendix b for details). an additional experiment involving the negation of congruent sentences is presented in appendix c. ## results as shown in the best performance for each llm across prompts is shown. models with the best overlapping accuracy are highlighted. we include results for a baseline where the models made a response to the conclusion only, and for a generative experiment where palm-540b freely generated responses to the questions. for romanian. we also observed strong performance in the 5-shot generative setting, at 87.67% for english and 80% for romanian. the largest models (palm-540b, flan-palm-540b and gpt-3-davinci) performed consistently better than the smaller models. interestingly, palm-540b greatly benefited from 5-shot prompting in romanian, whereas gpt-3-davinci showed slightly worse results with more shots. smaller gpt-3 models and palm-8b almost always performed close to chance level, whereas the other palm models benefited from few-shot prompts in english. we observed that some of the smaller models had consistent class bias, consistently answering "no" and thus scoring predominantly correctly on incongruent items only. the performance of the models on the baseline examples suggests that a small part of the performance can be explained by the likelihood of the conclusion only, and not just reasoning capacity. however, as in all baseline cases the performance does not approach that of the original examples, the likelihood of the conclusion is not sufficient to explain the performance of the models. the overall performance was better for the english than for the romanian dataset particularly in the case of palm models, including in the generative experiment. we expected this gap, in line with results from other multilingual tasks @xcite . as shown in ## conclusions we have introduced a novel and challenging benchmark for commonsense reasoning with spatial prepositions in multiple conceptual domains, and provided initial results on two families of llms. the task is part of our efforts towards investigating the limits of foundational reasoning in llms. our task captures highly variable performance scores across llms, with smaller llms typically performing at chance level and larger models approaching, but not reaching, human performance. the range of performance on this task makes it suitable as a checkpoint in examining trade-offs in models size and performance, particularly when complex or abstract reasoning is involved. we hope to encourage the development of more tasks that capture the building blocks of reasoning in llms.