Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generation (2022)

Abstract: Image-to-text tasks such as open-ended image captioning and controllable image description have received extensive attention for decades. Here we advance this line of work further, presenting Visual Spatial Description (VSD), a new perspective for image-to-text toward spatial semantics. Given an image and two objects inside it, VSD aims to produce one description focusing on the spatial perspective between the two objects. Accordingly, we annotate a dataset manually to facilitate the investigation of the newly-introduced task, and then build several benchmark encoder-decoder models by using VL-BART and VL-T5 as backbones. In addition, we investigate visual spatial relationship classification (VSRC) information into our model by pipeline and end-to-end architectures. Finally, we conduct experiments on our benchmark dataset to evaluate all our models. Results show that our models are awe-inspiring, offering accurate and human-like spatial-oriented text descriptions. Besides, VSRC has great potential for VSD, and the joint end-to-end architecture is the better choice for their integration. We will make the dataset and codes publicly available for research purposes.

## introduction text generation from images is a widely-adopted means for deep understanding of cross-modal data that has received increasing interest of both computer vision (cv) and natural language processing (nlp) communities @xcite . imageto-text tasks generate natural language texts to assist in understanding the scene meaning of a specific image, which might be beneficial for a variety of applications such as image retrieval @xcite , perception assistance @xcite , pedestrian detection @xcite , and medical system @xcite . ## related work image-to-text has been intensively investigated with the support of neural networks in the past years @xcite . the encoder-decoder architecture is an often considered framework, where the encoder extracts visual features from the image and the decoder generates text for specific tasks. early works employ a convolutional neural network (cnn) as the visual encoder and a recurrent neural network (rnn) as the text decoder @xcite . recently, the transformer neural network @xcite , which is impressively powerful in feature representation learning on both vision and language, has gained increasing interest. the transformer-based encoder-decoder models have been adopted in a wide range of image-to-text tasks @xcite @xcite . these models coupled with visuallanguage pretraining have achieved the top performance for these tasks @xcite @xcite @xcite @xcite . in this work, we exploit the transformer-based architecture and two pretrained visual-language models: vl-bart and vl-t5 @xcite , reaching several strong benchmark models for our task. image-to-text can be varied depending on the objective of the visual description. image captioning is the most well-studied task, which aims to summarize a given image or to describe a particular region in it @xcite . several subsequent studies have attempted to produce captions with specified patterns and styles @xcite @xcite @xcite . for example, vqa and visual reasoning can be regarded as such attempts, which are conditioned by a specific question directed at the input image @xcite @xcite . the vsr-guided image captioning @xcite is the most close to our work, which generates a sentence for a particular event in the image with well-specified semantic roles. here we focus on spatial semantics instead, generating a description based on the spatial relationship. spatial semantics is an important topic in both language and visual analysis. @xcite propose an preliminary study on text-based spatial role labeling. later, spatial element extraction and relation extraction from texts are investigated by @xcite . @xcite present a fine-grained spatial semantic analysis in texts with rich spatial roles. based on the image input, @xcite propose vsrc and benchmark it with a manually-crafted dataset. the vsrc is actually a shallow task for visual spatial analysis based on a closed relationship set and by using a simple classification schema. following, @xcite build a much stronger model on the dataset. many studies have exploited spatial semantics to assist other image understanding tasks @xcite @xcite @xcite . in addition, learning spatial representations from multiple modalities also receives particular attention @xcite . in this work, we extend image-to-text and propose vsd, which aims for the spatial understanding of the image. ## model we exploit the transformer-based encoder-decoder architecture to accomplish our vsd goal. the architecture can be partially pretrained from ultralarge-scale self-supervised datasets, which makes it capable of obtaining the top performance on a range of image-to-text generation tasks @xcite @xcite . in this section, we first briefly summarize the adopted model architecture and describe the two well-pretrained models exploited as backbones. ## conclusion in this work, we introduced a novel image-to-text generation task, namely vsd, aiming to generate text descriptions containing spatial semantics of two objects in an image, and constructed a dataset to benchmark this task. we adopted the models with transformer-based encoder-decoder architectures (i.e., vl-bart and vl-t5) for our task to obtain the baseline results. moreover, we proposed to integrate vsrc into our models by pipeline and end-to-end architectures, enhancing vsd with the support of spatial relations. the experimental results show that the vsrc-enhanced approach achieves significant progress over our initial models. moreover, the end-to-end models outperform the pipeline ones due to joint learning.