Sketchy Scene Captioning: Learning Multi-Level Semantic Information from Sparse Visual Scene Cues (2021)

Abstract: To enrich the research about sketch modality a new task termed Sketchy Scene Captioning isproposed in this paper. This task aims to generate sentence-level and paragraph-level descrip-tions for a sketchy scene. The sentence-level description provides the salient semantics of asketchy scene while the paragraph-level description gives more details about the sketchy scene. Sketchy Scene Captioning can be viewed as an extension of sketch classification which can onlyprovide one class label for a sketch. To generate multi-level descriptions for a sketchy scene ischallenging because of the visual sparsity and ambiguity of the sketch modality. To achieve ourgoal we first contribute a sketchy scene captioning dataset to lay the foundation of this new task. The popular sequence learning scheme e.g. Long Short-Term Memory neural network with vi-sual attention mechanism is then adopted to recognize the objects in a sketchy scene and inferthe relations among the objects. In the experiments promising results have been achieved on the proposed dataset. We believe that this work will motivate further researches on the understanding of sketch modality and the numerous sketch-based applications in our daily life. The collected dataset is released at https://github.com/SketchysceneCaption/Dataset .

## introduction in recent years, sketch has emerged as one important data modality @xcite . compared to a natural image, a sketch only contains sparse and ambiguous visual information. current works about sketch mainly focus on predicting one class label for a sketch, and such a label provides very limited semantic information @xcite . differently, the tasks about natural image are abundant, such as classification @xcite , captioning @xcite @xcite , and visual question answering @xcite . what hinders the research about sketch is the lack of sketch datasets. specifically, natural images are easy to be obtained and a lot of efforts have been put into annotating the images. by contrast, sketch is created by human and the generation of a sketch is time-consuming, which limits the volume of a sketch dataset and the visual details of the sketches in the dataset. hence, most of current sketch datasets only contain sketches with a single object and the corresponding class label. drawing inspiration from the task of natural image captioning, it is attractive to expand a sketch to a sketchy scene which contains several objects, and extend the class label to a sentence-level or even a paragraph-level description. in a word, a sketchy scene dataset with multi-level descriptions is in urgent need to promote the research about sketch. one promising application with above extension is child education @xcite . specifically, with the wide popularity of tablet pc, it becomes common for a child to doodle on a touch screen. to interact with a child, a computer agent needs to understand what a child has drawn and give reasonable response to the child. for example, if a sketchy scene drawn by a child cannot match the sentence or paragraph given by the agent, the child is required to draw the sketchy scene again, which helps improve the drawing skill of the child. another potential application is the assistance for the visually impaired computational linguistics people. with simple and sparse visual content, a sketchy scene can be easily turned into a concaveconvex plate that can be read by a visually impaired person in a touch manner. with the corresponding caption transformed into human voice @xcite , the visually impaired person can feel what are depicted in the sketchy scene without the help of others. other potential applications include largescale sketchy scene retrieval via human language and automatic sketch management on the web (e.g., to cluster the numerous sketchy scenes with similar topics). motivated by the observations above, we extend the task of sketch classification to sketchy scene captioning, a task that aims to generate multi-level (i.e., sentence-level and paragraph-level) descriptions for a sketchy scene, as shown in figure 1 . to the best of our knowledge, our work is the first attempt to generate multi-level and dense descriptions for a sketchy scene. currently, to achieve the goal of sketchy scene captioning is very challenging for two reasons. first, compared to a natural image captioning dataset, only the generation of a sketchy scene is time-consuming, let alone the annotation of the sketchy scene. second, a sketchy scene only contains very sparse visual cues. that is, an object is depicted only with some lines. in addition, the visual cues of a sketchy scene are also ambiguous. that is, the objects in different sketchy scenes have great variations in appearance, making it difficult to distinguish the objects. to overcome the above two challenges, we create a sketchy scene dataset with multi-level descriptions and achieve the goal of sketchy scene captioning using several sequence-learning-based models in the field of image captioning @xcite @xcite . the contributions of this work are three-folds: 1) a new task termed sketchy scene captioning is proposed to generate multilevel descriptions for a sketchy scene. this task can be treated as a new paradigm for comprehensive understanding of the sparse visual cues; 2) a sketchy scene captioning dataset is constructed based on sketchyscene dataset @xcite . currently, the new dataset contains 1,000 sketchy scenes with both the sentence-level and paragraph-level captions; and 3) promising experimental results have been achieved on the newly collected dataset, demonstrating the potentials of sketchy scene captioning. we hope this work could help motivate further researches on mining multi-level semantic information from sketchy scenes. ## related works in this section, we will briefly review the related works about sketch and image captioning. the differences between the prior works and ours will be discussed as well. current works about sketch mainly focus on sketch classification and sketch-based image retrieval (sbir). sketch classification is a task of recognizing what object is depicted in a sketch. sbir aims to retrieve a natural image for a given query sketch. in recent years, great progresses have been made in the field of sketch. for example, @xcite proposed sketch-a-net, a multi-scale and multi-channel deep neural network, to yield the sketch recognition performance surpassing that of humans on the tu-berlin sketch dataset @xcite . @xcite proposed the sketchy dataset, which c c l 2 0 2 1 computational linguistics was the first large-scale collection of sketch-photo pairs for image retrieval. @xcite proposed a deep visual-sequential fusion mechanism to model the visual and sequential patterns of the strokes of a sketch. @xcite proposed a semantic-aware knowledge preservation method for sketch-based image retrieval. in spite of the above progresses, the related works about sketch classification are limited to assigning a class label to each sketch. in this paper, we go a step further to generate multi-level and dense descriptions for a sketchy scene. current methods for image captioning can be mainly divided into three categories, that is, templatebased, retrieval-based, and sequence-learning-based. in the template-based method, the salient objects, their attributes, and the relations among objects in an image were first recognized, and a pre-defined template was then filled with the detected information to yield a full sentence @xcite . the retrieval-based method first obtained the visually similar image with the query image, and then used the description of the retrieved image as the description of the query image @xcite . however, these two methods could only generate relatively fixed sentences, relying on the given imagecaption dataset. in the era of deep learning, sequence learning was adopted to adaptively generate a description for a natural image, where a convolutional neural network (cnn) @xcite encoder was used to encode the image into a high-level visual representation, and a recurrent neural network (rnn) @xcite decoder was adopted to "translate" the image representation into a sentence. typically, @xcite first proposed to use inception @xcite convolutional neural network as the encoder to convert an image into a fixed-length vector, and then use long short-term memory (lstm) @xcite neural network as the decoder to generate a caption for the image. @xcite introduced two spatial visual attention mechanisms to help the model dynamically focus on the image regions corresponding to the word that was about to be generated. besides, @xcite designed a hierarchical lstm model to generate a paragraphlevel description for a natural image. overall, current works on image captioning mainly focus on natural images. differently, we explore the caption generation problem in the field of a different domain, that is, sketchy scene which only contains sparse and ambiguous visual information. ## a new dataset for sketchy scene captioning to the best of our knowledge, there is no available dataset for sketchy scene captioning. hence, we need to first construct a sketchy scene dataset with sentence-level and paragraph-level descriptions. next, we will describe how the dataset is collected in details. ## multi-level sketchy scene captioning through sequence learning in this work, the popular sequence-learning-based method is adopted for flexible sketchy scene captioning, as shown in figure 1 . our framework integrates sketchy scene encoder for deep visual features (i.e., encoding a sketchy scene at an abstract level to obtain a discriminative visual representation) and sketchy scene decoder with spatial visual attention (i.e., grasping more visual details of a sketchy scene while generating the description). it is a new attempt to generate multi-level descriptions for a sketchy scene through the sequence learning paradigm. ## conclusion and future work in this paper, a new task termed sketchy scene captioning is proposed. this task aims to generate multilevel descriptions for a sketchy scene through the sequence learning paradigm. to achieve the goal, a new dataset consisting of 1,000 sketchy scenes with the corresponding sentence-level and paragraphlevel captions is created. the experimental results show that our captioning models can recognize the main objects in a sketchy scene and the interactions among these objects. this proves that it is feasible to generate multi-level captions for a sketchy scene. in the future, we plan to increase the volume of the dataset and explore how to learn a better representation of a sketchy scene for caption generation. we hope this work can inspire further researches on the better understanding of sketchy scenes.