Integrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization (2021)

Abstract: While much research has been done in text-to-image synthesis, little work has been done to explore the usage of linguistic structure of the input text. Such information is even more important for story visualization since its inputs have an explicit narrative structure that needs to be translated into an image sequence (or visual story). Prior work in this domain has shown that there is ample room for improvement in the generated image sequence in terms of visual quality, consistency and relevance. In this paper, we first explore the use of constituency parse trees using a Transformer-based recurrent architecture for encoding structured input. Second, we augment the structured input with commonsense information and study the impact of this external knowledge on the generation of visual story. Third, we also incorporate visual structure via bounding boxes and dense captioning to provide feedback about the characters/objects in generated images within a dual learning setup. We show that off-the-shelf dense-captioning models trained on Visual Genome can improve the spatial structure of images from a different target domain without needing fine-tuning. We train the model end-to-end using intra-story contrastive loss (between words and image sub-regions) and show significant improvements in visual quality. Finally, we provide an analysis of the linguistic and visuo-spatial information.

## introduction story visualization is an emerging area of research with several potentially interesting applications such as visualization of educational materials, assisting artists with web-comic creation etc. each story consists of a sequence of images along with a sequence of captions describing the content of the images. the goal of the task is to reproduce the images given the captions. it is more challenging than conventional text-to-image generation @xcite because the generative model needs to identify the narrative structure expressed in the sequence of captions and translate it into a story of images. some critical features of a good story include consistent character and background appearances, relevance to individual captions as well as overall story, and coherent narrative. while recent text-to-image models @xcite @xcite are successfully generating high-quality images, they are not directly designed for narrative understanding over sequential text. hence, story visualization necessitates independent research towards developing generative models for the task. in this paper, we explore the use of visuo-linguistic structured inputs and outputs for improving story visualization. towards this end, we propose (v)isuo-spatial, (l)inguistic & (c)ommonsense i.e. vlc-storygan which (1) uses constituency parse trees and commonsense knowledge as input using structure-aware encoders, (2) leverages a pretrained dense captioning model for additional position and semantic information, and (3) is trained using intra-story contrastive loss for maximizing global semantic alignment between input captions and generated visual stories. grammatical structures like constituency parse trees are potentially rich sources of information for visualizing relations between objects (or characters), their actions, and their attribute (property) modifiers. @xcite ; @xcite ; @xcite ; @xcite demonstrate that inducing such tree-structures within the encoder guides words to compose the meaning of longer phrases hierarchically and improves various tasks like masked language modeling, translation, visual grounding of language etc., suggesting potential gains in other tasks. most text-to-image synthesis as well as story visualization models @xcite perform flat processing over free-text captions using lstm or transformer-based encoders. hence, in order to leverage the grammatical information packed in constituency parse trees, we propose a novel memory-augmented recurrent tree-transformer (martt) to encode captions and promote forward flow of hierarchical information across the sequence of captions for each story. further, we find that input captions in story visualization lack details about the visual elements in the image. hence, we augment the captions with external knowledge. for instance, when one caption mentions snow while the other mentions icy roads, we provide the knowledge that both are related to cold weather, encouraging the model to learn similar representations for either of the phrases. dual learning has served as an effective method for promoting desirable characteristics in target output for both text-to-image generation @xcite and story visualization @xcite . @xcite use image segmentation to preserve character shapes while @xcite use video captioning for global alignment between the input caption and the generated sequence of frames. each of these auxiliary tasks generate uni-modal outputs, dealing either with image or text. in a bid to combine the benefits of learning signals from both visuo-spatial and language modalities, we propose the use of dense captioning as the dual task, which has proven useful as a source of complementary information for many vision-language tasks @xcite @xcite . dense captioning models provide regional bounding boxes for objects in the input image and also describe the region. by using these outputs for dual learning feedback for story visualization, the generative model receives a signal rich in spatial as well as semantic information. the spatial signal is especially important for our task since the input captions do not contain any specifications about the shape, size or position of characters within the story. further, we find that off-the-shelf dense captioning models, which are trained on realistic images from visual genome, transfer well to a markedly different domain like cartoon and can be used to provide visuo-spatial feedback without finetuning on target domain. finally, we want the model to recognize the subtle differences between frames in a story and generate relevant images that fit into a coherent narrative. hence, we employ contrastive loss between imageregions and words in the captions at each timestep to improve semantic alignment between the caption and image. adjacent frames in a story often contain subtle differences, as can be seen in an example in fig. 1 . we modify the region-word contrastive loss proposed in @xcite for story visualization by sampling negative images from adjacent frames, forcing the model to recognize the difference between frames. overall, our contributions are: (1) we propose vlc-storygan to use linguistic information, augmented with commonsense knowledge, for conditional image synthesis. (2) use dense-captioning to provide complementary positional and semantic information during training and show that off-the-shelf models trained on visual genome can be effective without fine-tuning on the target domain. story understanding & commonsense. @xcite introduced relationship modelling networks to extract evolving relationship trajectories between two characters in a novel. @xcite use latent variables to weigh predefined semantic aspects like topical consistency to improve encoding for story completion. @xcite ; @xcite augment story encodings with structured commonsense knowledge to improve story ending generation. we focus on the use of structured commonsense as well as grammatical trees to improve story encoding for the end goal of visualization. tree encoder. tree structures have traditionally been encoded using tree lstms @xcite @xcite . in recent work, @xcite enforce a hierarchical prior in the self-attention layer of transformer @xcite and @xcite use a parent-sibling tree convolution block to perform structure-aware encoding. @xcite use sub-tree masking and hierarchical accumulation to improve machine translation. we propose a simpler tree-transformer architecture, augmented with memory units @xcite for recurrence. contrastive loss. @xcite first proposed the contrastive loss in text-to-image synthesis through the deep attentional multimodal similarity model (damsm). contragan @xcite performs minimization of contrastive loss between multiple image embeddings in the same batch, in addition to class embeddings (miy-ato and koyama, 2018). @xcite combine inter-modality and intra-modality contrastive losses and observe complementary improvements. we adapt inter-modal loss for story visualization by sampling negatives from adjacent frames. dense captioning. dense captioning jointly localizes semantic regions and describes these regions with short phrases in natural language @xcite . @xcite and @xcite use dense captions for visual and video question answering respectively. we use a pretrained dense captioning model to first annotate our target dataset and then use it within a dual learning framework to improve image synthesis for story visualization. ## experimental settings evaluation. we adopt the metrics proposed in @xcite al. (2019c) and maharana et al. (2021): • character classification: frame accuracy (exact match) and classification f1-score using finetuned inception-v3 to measure visual quality of recurring characters in predicted images. @xcite . since story visualization datasets are adapted from a video captioning dataset, sometimes a single frame does not represent the caption perfectly. however, during training, we sample a frame from the video every time, thus providing coverage for the entire video and association between all characters in the story and their representation in the frame. with this process, the model is able to observe all characters from the caption in the target frames during training time. during inference, our target is a static story, and not a video. hence, we evaluate the predictions under the assumption that all characters should appear in the frame. dataset. we use the pororosv dataset proposed in @xcite , and the splits proposed in @xcite to evaluate our approach. each sample in pororosv has 5 frames and 5 corresponding captions that form a narrative. there are 9 recurring characters throughout the dataset. each character is featured in at least 10% of the frames, making it crucial for the model to be capable of generating each of them. there are 10191/2334/2208 samples in training, validation and test splits respectively. the constituency parses are extracted and pre-processed using spacy @xcite and nltk @xcite . 3 for commonsense knowledge, we first extract nouns and verb words from all of the captions in a story, and find conceptnet triples @xcite containing at least one of those words in the subject and object phrases. next, we use pretrained glove embeddings @xcite to find a broader pool of words which are related to the words and find additional relevant triples. these triples are combined into knowledge graph inputs for each frame. we use the top ten bounding box and caption predictions from a dense captioning model pretrained on visual genome @xcite for dual learning. experiments. our model is developed using py-torch. all models are trained on the proposed training split and evaluated on validation and test sets. we select the best checkpoints and tune hyperparameters by using the character classification f-score on the validation set. ## analysis and discussion in this section, we take a closer look at the various data sources for vlc-storygan. ## conclusion in this paper, we investigate the use of structured knowledge for the task of story visualization. we propose a novel recurrent tree-transformer for encoding constituency trees and augment it with commonsense knowledge. we train the model using dense captioning loss and intra-story contrastive loss. our results demonstrate the effectiveness of these approaches. we believe that these methods will encourage the use of structured knowledge for story visualization and text-to-image synthesis.