Things not Written in Text: Exploring Spatial Commonsense from Visual Signals (2022)

Abstract: Spatial commonsense, the knowledge about spatial position and relationship between objects (like the relative size of a lion and a girl, and the position of a boy relative to a bicycle when cycling), is an important part of commonsense knowledge. Although pretrained language models (PLMs) succeed in many NLP tasks, they are shown to be ineffective in spatial commonsense reasoning. Starting from the observation that images are more likely to exhibit spatial commonsense than texts, we explore whether models with visual signals learn more spatial commonsense than text-based PLMs. We propose a spatial commonsense benchmark that focuses on the relative scales of objects, and the positional relationship between people and objects under different actions. We probe PLMs and models with visual signals, including vision-language pretrained models and image synthesis models, on this benchmark, and find that image synthesis models are more capable of learning accurate and consistent spatial knowledge than other models. The spatial knowledge from image synthesis models also helps in natural language understanding tasks that require spatial commonsense.

## introduction spatial perception, the ability to detect the spatial position and to infer the relationship between visual stimuli @xcite , is basic but important for human beings @xcite . it is of everyday use, from understanding the surrounding environment, like when seeing a woman sitting in a car with her hands on the steering wheel, we know she is probably driving, to processing spatial information and performing reasoning, like navigating ## solving natural language questions we investigate how to acquire spatial knowledge from isms and whether the knowledge is effective in natural language understanding scenarios. to our best knowledge, there is no appropriate task that focuses on spatial commonsense, so we create a toy task by transforming our probing benchmark into the form of question answering (qa). dataset. we construct a qa dataset of yes/no questions. questions of objects' sizes are in the form of is o a larger/smaller than o b ? and questions of objects' heights are like is o a taller/shorter than o b ?, where o a and o b are two objects. questions about positional relationship are accompanied with the action: for instance, a man washes the car. is the man inside the car? to avoid bias in answer distribution, the numbers of yes and no are equal in gold answers. there are 500 questions for size, 500 for height, and 448 for positional relationship. models. we use vinvl-base together with our image synthesis model vqgan+clip to answer spatial commonsense questions. the vinvl here is finetuned on the vqa @xcite task. it takes images generated from ism with textual prompts from questions, and predicts the answer based on the question and image together. note that the vqa training corpus does not contain commonsense reasoning questions. we choose unifiedqa @xcite as a text-based qa model for comparison. based on the pretrained t5 model @xcite , uni-fiedqa is continually trained on various qa tasks, including three yes/no datasets. we use unifiedqalarge, which is comparable with our synthesis and reasoning model (ism w/ vinvl) in size. results. as shown in ## conclusion we propose a new spatial commonsense probing framework to investigate object scales and positional relationship knowledge in text-based pretrained models and models with visual signals. experimental results show that models with visual signals, especially isms, learn more accurate and consistent spatial commonsense than text-only models. integrating isms with visual reasoning models outperforms plms in answering spatial questions. this manifests the potential of using spatial knowledge from isms in natural language understanding tasks.