Diverse and Relevant Visual Storytelling with Scene Graph Embeddings (2020)

Abstract: A problem in automatically generated stories for image sequences is that they use overly generic vocabulary and phrase structure and fail to match the distributional characteristics of human-generated text. We address this problem by introducing explicit representations for objects and their relations by extracting scene graphs from the images. Utilizing an embedding of this scene graph enables our model to more explicitly reason over objects and their relations during story generation, compared to the global features from an object classifier used in previous work. We apply metrics that account for the diversity of words and phrases of generated stories as well as for reference to narratively-salient image features and show that our approach outperforms previous systems. Our experiments also indicate that our models obtain competitive results on reference-based metrics.

## introduction visual storytelling is the generation of a coherent narrative from a series of images @xcite . in this paper, we address a particular challenge in visual storytelling: reflecting human preferences in narrative structure, especially the choice of content words and phrases that comprise a readable story. humans prefer to use diverse words and phrases to construct the storyline to avoid repetitions within or across sentences. for example, in the human-written story in fig. 1 , very few content words are repeated. however, @xcite have found that recent work often generate repetitive words and phrases which leads to repetitions across sentences and makes stories less diverse. for example, in the first story of fig. 1 , the model generates a verb phrase had a great time and then repeats it in the fifth sentence. these words 1 typo generated by human: "have" instead of "gave". ## related work visual storytelling. extracting a good representation of the information in the visual input is a key part of the visual storytelling task. prior work in visual storytelling has typically opted for global features extracted from a pre-trained convolutional neural network @xcite @xcite and has focused on improving the language generation model. @xcite show that introducing regional features and implicit coreference relations of entities leads to more human-realistic word usage in generated stories. only few prior works employ an intermediate structured representation on story telling task. @xcite use an external database of knowledge graphs to enchance the visual representation and improve story telling performance. we use scene graphs extracted from images, which does not require an external knowledgebase. @xcite extract scene graphs from images and train an end-to-end model with a graph convolutional encoder directly on visual stories. we propose a pipeline method which first obtains scene graph embeddings from images then applies them to visual storytelling in order to reduce the difficulty of learning both the scene graph embeddings and the story generation model together. our attention-based story generation model has less parameters while obtaining competitive results. scene graph representation. a scene graph is a symbolic representation of structural information where entities are nodes and their relations are edges @xcite . the large scenegraph annotated visual genome @xcite dataset has enabled the development of models to extract scene graph representations from images @xcite . these scene graph represenations have proven effective on various tasks like image retrieval @xcite and image generation @xcite . scene graph based image captioning. a sequential scene graph representation is used to encode images in @xcite to improve image captioning. @xcite propose auto-encoding text-based scene graphs to learn a shared dictionary between visual and text based graphs, achieving state-of-the-art image captioning performance. @xcite show that image scene graphs extracted using a trained model can match the captioning performance of an oracle with access to ground-truth graphs. aligning text-and imagebased scene graphs has also been used to generate image captions without paired data @xcite . ## model design the task of visual storytelling can be decomposed into two distinct parts: (1) extracting relevant information from input images i into compact features and (2) generating stories using these visual features. we improve the visual feature representation by switching from commonly-used global feature vectors to a scene graph-based representation which explicitly encodes objects and their relations. we also reduce the number of parameters by taking a modular approach that separates learning scene graph embeddings from images and generating visual stories. this allows us to independently train the scene graph embedding model and to design a visual storytelling model with fewer parameters yet competitive performance. our full pipeline is shown in fig. 2 . we first apply a scene graph generator to extract scene graphs containing vertices for objects and edges for relations between two objects. we then augment the scene graph for each image by adding regional features (see section 3.1). a graph neural network embeds each graph node by aggregating information from across the graph. we propose a pre-training step to independently learn this graph embedding. to do this, we obtain the confidence of the object detector for each object in each image, termed as visual saliency, and construct a sequence of object labels ordered by their visual saliency for each image. then we train a graph-to-sequence model to predict this object sequence given the scene graph embedding of the corresponding image (see section 3.2). to generate stories, we extract both global and regional features from the scene graph embedding for each image and feed them to an attention-based story generation model (see section 3.3). ## experiment and evaluation now we show that using pre-trained scene graph embeddings yields competitive results as compared to state-of-the-art approaches on reference-based metrics while using fewer parameters in the image encoder. we also perform an ablation study to show that all proposed components contribute to the performance of the full model and that scene graph embeddings are effective across different attention mechanisms. while the reference-based metrics are useful, they do not always correlate with better story quality as perceived by humans @xcite . hence, we also evaluate our model in terms of diversity of word and phrase structure and propose metrics to explicity measure the correctness of object references in section 5. results show that our scene graph-based model uses more diverse/relevant words and phrases compared to prior work. ## evaluating diversity and relevance to get an in-depth understanding of the diversity of different types of words or phrases in generated stories, we perform the first fine-grained analysis of the distributions of words by different part-of-speech (pos) tags and phrases by different constituent tags. we first process the generated stories with a state-of-the-art pos tagger and constituency parser @xcite . then we plot the frequency vs. rank distributions following zipf's law for each pos tag and each constituent tag. we follow @xcite to compute the zipf's coefficient to check how similar the distributions of generated stories are to human-written stories. using this metric, we compare the diversity of output stories from our model to the baselines and to the best-available prior work, arel foot_0 . ## conclusions we show that introducing scene graph embeddings into visual storytelling with a pipeline method can obtain competitive results while reducing the number of parameters in the storytelling model. we also perform the first fine-grained analysis on the distributions of words and phrases in generated stories which shows that scene graph embeddings increase word and phrase diversities and bring the distributions closer to that of humans. we finally show that the diverse noun phrases we generate are more relevant to the objects in the images. future work one benefit of this work is that it provides a baseline for the pre-training of images in visual storytelling, allowing for any images to be used to augment the model without requiring story text; in future work, we will show that this mitigates the limitation of data size. we are currently working on how to merge regional representations for each graph effectively in pre-training and storytelling. gcn is a powerful method for pre-training, but the number of layers is strongly related to the diameter of the graph which is highly variable. a solution is to use graph transformer @xcite which learns global attentions across the whole graph. moreover, we would like to explore how to extract features from images more accurately for storytelling. the edges of scene graphs in the visual genome dataset only contain spatio-temporal relations and limited numbers of general actions like 'holding' as in fig. 1 . we need to extract more common-sense directed events like 'giving' from a sub-graph of the scene graph. this requires implicit graph induction in the current model; we will test an explicit component.