Generating Visual Spatial Description via Holistic 3 D Scene Understanding (2023)

Abstract: Visual spatial description (VSD) aims to generate texts that describe the spatial relations of the given objects within images. Existing VSD work merely models the 2D geometrical vision features, thus inevitably falling prey to the problem of skewed spatial understanding of target objects. In this work, we investigate the incorporation of 3D scene features for VSD. With an external 3D scene extractor, we obtain the 3D objects and scene features for input images, based on which we construct a target object-centered 3D spatial scene graph (Go3D-S2G), such that we model the spatial semantics of target objects within the holistic 3D scenes. Besides, we propose a scene subgraph selecting mechanism, sampling topologically-diverse subgraphs from Go3D-S2G, where the diverse local structure features are navigated to yield spatially-diversified text generation. Experimental results on two VSD datasets demonstrate that our framework outperforms the baselines significantly, especially improving on the cases with complex visual spatial relations. Meanwhile, our method can produce more spatially-diversified generation.

## introduction visual spatial description is a newly emerged vision-language task, which aims to generate a textual descriptive sentence of the spatial relationship between two target visual objects in a given image @xcite . vsd falls into the category of image-to-text generation, while in particular focusing on the visual spatial semantics understanding, which has great values on the real-world humancomputer interaction (hci) applications @xcite , e.g., automatic navigation @xcite , personal assistance @xcite , and unmanned manipulation @xcite @xcite ). ## methodology problem definition given an image i with two object @xmath0, o 2 > in i, vsd generates a sequence of words @xmath1, ..., w n } that describes the spatial relationship between o 1 and o 2 . the input o 1 and o 2 contain the object tags and their 2d location coordinates. different from image captioning, the generated sentences of vsd must directly or indirectly express the spatial relation between the target objects. overall framework as shown in figure 2 , our framework (namely 3dvsd) is built upon an encoder-decoder paradigm, where the encoder is responsible for the 3d scene feature modeling, and the decoder generates spatially-diversified descriptions based on the spatial semantic features learned from encoder. ## related work image-to-text (i2t) is a fundamental task category of the vision-language multimodal topic. existing i2t tasks, e.g., image captioning @xcite @xcite and vqa @xcite @xcite , attempt to generate textual pieces to understand the image content semantics through different perspectives. vsd is also a subtask of i2t, which however places the focus on the spatial semantics understanding. within recent years, vl-ptms are extensively employed for the i2t tasks, which have helped achieve state-of-theart performances on many benchmarks @xcite @xcite @xcite . prior vsd study @xcite has benchmarked the vsd task with these general-purpose vl-ptms. unfortunately, different from the content-semantic i2t tasks, the goal of vsd is to grasp the spatial semantics. this work also relates to the theme of visual spatial understanding, which has been intensively investigated in the past years. @xcite ; @xcite propose the visual spatial relation classification (vsrc) task, aiming to predict whether a spatial description is reasonable for the input image. differently, vsd aims to directly generate the descriptions of the spatial relations. the bottleneck of visual spatial understanding tasks is the capturing of spatial semantics. thus, in this paper, we propose improving the visual spatial understanding for vsd via modeling the holistic 3d scene features. our work takes the advancements from the topic of 3d scene parsing @xcite @xcite , which reconstructs the 3d scene from a single rgb image. by using the 3d scene features, e.g., 3d layout and 3d object, we achieve the goal of more in-depth understanding of the spatial semantics. ## conclusion in this work we incorporate the 3d scene features for improving the visual spatial description (vsd) task. we first employ an off-the-shelf 3d scene extractor to produce 3d objects and scene features for the input images. we then build a target object-centered 3d spatial scene graph (go3d-s 2 g) structure, which is encoded with an objectcentered graph convolutional network. next, we devise a scene subgraph selecting mechanism to sample topologically-diverse subgraphs from go3d-s 2 g, where the local features are used to guide to generate spatially-diversified descriptions. on the vsd datasets our framework shows superiority on the task, especially for solving the complex cases, such as layout-overlapped and irregularly-posed object. meanwhile our method can produce more spatially-diversified generation. finally, we demonstrate the influence of quality of the external 3d scene extractor.