D ecoding Language Spatial Relations to 2 D Spatial Arrangements (2020)

Abstract: We address the problem of multimodal spatial understanding by decoding a set of language-expressed spatial relations to a set of 2D spatial arrangements in a multi-object and multi-relationship setting. We frame the task as arranging a scene of clip-arts given a textual description. We propose a simple and effective model architecture Spatial-Reasoning Bert (SR-Bert), trained to decode text to 2D spatial arrangements in a non-autoregressive manner. SR-Bert can decode both explicit and implicit language to 2D spatial arrangements, generalizes to out-of-sample data to a reasonable extent and can generate complete abstract scenes if paired with a clip-arts predictor. Finally, we qualitatively evaluate our method with a user study, validating that our generated spatial arrangements align with human expectation.

## introduction spatial understanding is a problem of paramount importance to both the vision and the language community. for a machine learning model to be able to reason about the spatial domain w.r.t. another modality (language, vision, etc.), it should incorporate common-sense spatial knowledge, which is often obvious to humans, yet hard to grasp by machines. if the spatial relations are expressed in language, such common sense knowledge can be hidden within, e.g., "mike and jenny see a duck" (figure 1 ) -meaning that both "mike" and "jenny" should be facing the "duck". the complexity of the problem increases tremendously when there is no limit on the number/type of objects and relationships -the de-facto setting in computer graphics, video games, 3d modelling, etc. communicating spatial relations via language is intuitive for humans, while arranging objects in a multidimensional space is tedious. therefore, building jenn ha a hamb ge . mike i anding nde a helico e . mike and jenn a e ha . mike and jenn a e anding ne o he able. ## dataset we use the abstract scenes dataset (zitnick and parikh, 2013) which consists of 10.020 scenes of clip-arts, together with ⇠6 sentences for each scene, describing the scene content and spatial relations between them. the clip-arts belong to 7 distinct groups, namely objects in the sky, large elements, people, animals, clothing, food and toys. the scenes are organized in 1002 semantically different sets, where scenes within a particular set are generated from the same core-scene description. after removing empty scenes, from each of the 1002 sets we allocate one scene for testing, one for model selection and we keep the rest for training. 2 that leaves us with 1002 scenes in the test and validation set respectively, and 7989 scenes in the training set. the maximum number of clip-arts in a scene is 17 while the minimum and median are 6. the total number of unique clip-arts in the dataset is 126. ## methods the main building block for all our models is bert @xcite . in particular, we present sr-bert, a bert variant based on a pre-trained bert base . compared with existing bert architectures @xcite @xcite @xcite , with sr-bert our contributions are twofold: (i) we alter the input-embedding module to process two discrete modalities with a different positional encoding -sequential and spatial. (ii) we design a novel training method -masked position modelling, where we iteratively mask and predict the positional encoding of the input tokens. ## conclusion in this paper, we address the problem of spatial understanding by predicting spatial arrangements of scenes given their natural language descriptions. this work advances towards general spatial understanding of visual scenes and language by addressing the limitations of prior work: (i) modelling only pairwise relationships; (ii) using scene environments with strong priors on (relative) object placements (e.g., indoor home environments); (iii) use of structured language (instead of natural language). we proposed a novel architecture -sr-bert, for which we empirically demonstrate that it is capable of reasoning about an arbitrary number of objects and the relationships between them in the 2d space. sr-bert's spatial reasoning is irrespective of the spatial language type employed (explicit or implicit) and effectively generalizes to out-of-sample instances -a frequently occurring real-world situation where one encounters novel (spatial) scene descriptions. the first limitation of our approach is that it is restricted to 2d spatial reasoning despite computer graphics data being prevalently 3d. however, we kept a simple setting for our first study of spatial reasoning while our approach can trivially be extended to work with 3d data. second, we assume a fixed number of 126 clip-art categories and canvas of 500x400, which is a rather idealized setting considering that actual 2d/3d modelling is performed with: (1) larger number of categories, and (2) bigger canvases. finally, scene layout generation is an ill-posed problem, i.e., there are multiple valid layouts conditioned on a scene description. therefore, an ideal approach should not generate a deterministic layout, but rather address the uncertainty in both the model's output and the evaluation.